User Inputs

output.var = params$output.var 

transform.abs = FALSE
log.pred = params$log.pred
norm.pred = FALSE
algo.forward.caret = params$algo.forward.caret
algo.backward.caret = params$algo.backward.caret
algo.stepwise.caret = params$algo.stepwise.caret
algo.LASSO.caret = params$algo.LASSO.caret
algo.LARS.caret = params$algo.LARS.caret
message("Parameters used for training/prediction: ")

## Parameters used for training/prediction:

str(params)

## List of 7
##  $ output.var         : chr "y3"
##  $ log.pred           : logi TRUE
##  $ algo.forward.caret : logi TRUE
##  $ algo.backward.caret: logi TRUE
##  $ algo.stepwise.caret: logi TRUE
##  $ algo.LASSO.caret   : logi TRUE
##  $ algo.LARS.caret    : logi TRUE

# Setup Labels
output.var.tr = if (log.pred == TRUE)  paste0(output.var,'.log') else  output.var.tr = output.var

Loading Data

feat  = read.csv('../../Data/features_highprec.csv')
labels = read.csv('../../Data/labels.csv')
predictors = names(dplyr::select(feat,-JobName))
data.ori = inner_join(feat,labels,by='JobName')
#data.ori = inner_join(feat,select_at(labels,c('JobName',output.var)),by='JobName')

Data validation

cc  = complete.cases(data.ori)
data.notComplete = data.ori[! cc,]
data = data.ori[cc,] %>% select_at(c(predictors,output.var,'JobName'))
message('Original cases: ',nrow(data.ori))

## Original cases: 10000

message('Non-Complete cases: ',nrow(data.notComplete))

## Non-Complete cases: 3020

message('Complete cases: ',nrow(data))

## Complete cases: 6980

summary(dplyr::select_at(data,c('JobName',output.var)))

##       JobName           y3        
##  Job_00001:   1   Min.   : 95.91  
##  Job_00002:   1   1st Qu.:118.29  
##  Job_00003:   1   Median :124.03  
##  Job_00004:   1   Mean   :125.40  
##  Job_00007:   1   3rd Qu.:131.06  
##  Job_00008:   1   Max.   :193.73  
##  (Other)  :6974

Output Variable

The Output Variable y3 shows right skewness, so will proceed with a log transformation

Histogram

df=gather(select_at(data,output.var))
ggplot(df, aes(x=value)) + 
  geom_histogram(aes(y=..density..),bins = 50,fill='light blue') + 
  geom_density()

  #stat_function(fun = dnorm, n = 100, args = list(mean = mean(df$value), sd = sd(df$value)))

QQPlot

ggplot(gather(select_at(data,output.var)), aes(sample=value)) + 
  stat_qq() + 
  facet_wrap(~key, scales = 'free',ncol=4)

Trasformation of Output Variable from y3 to y3.log

if(log.pred==TRUE) data[[output.var.tr]] = log(data[[output.var]],10) else
  data[[output.var.tr]] = data[[output.var]]
df=gather(select_at(data,c(output.var,output.var.tr)))
ggplot(df, aes(value)) + 
  geom_histogram(aes(y=..density..),bins = 50,fill='light blue') + 
  geom_density() + 
  # stat_function(fun = dnorm, n = 100, args = list(mean = mean(df$value), sd = sd(df$value)))  
  facet_wrap(~key, scales = 'free',ncol=2)

ggplot(gather(select_at(data,c(output.var,output.var.tr))), aes(sample=value)) + 
  stat_qq() + 
  facet_wrap(~key, scales = 'free',ncol=4)

Best Normalizator y3

Normalization of y3 using bestNormalize package. (suggested orderNorm) This is cool, but I think is too far for the objective of the project

t=bestNormalize::bestNormalize(data[[output.var]])

## Best Normalizing transformation with 6980 Observations
##  Estimated Normality Statistics (Pearson P / df, lower => more normal):
##  - No transform: 2.9625 
##  - Box-Cox: 1.4152 
##  - Log_b(x+a): 2.0249 
##  - sqrt(x+a): 2.4466 
##  - exp(x): 749.2827 
##  - arcsinh(x): 2.0256 
##  - Yeo-Johnson: 1.1673 
##  - orderNorm: 1.1755 
## Estimation method: Out-of-sample via CV with 10 folds and 5 repeats
##  
## Based off these, bestNormalize chose:
## Standardized Yeo-Johnson Transformation with 6980 nonmissing obs.:
##  Estimated statistics:
##  - lambda = -1.998639 
##  - mean (before standardization) = 0.5003083 
##  - sd (before standardization) = 5.108542e-06

qqnorm(data[[output.var]])

qqnorm(predict(t))

orderNorm() is a rank-based procedure by which the values of a vector are mapped to their percentile, which is then mapped to the same percentile of the normal distribution. Without the presence of ties, this essentially guarantees that the transformation leads to a uniform distribution

Predictors

All predictors show a Fat-Tail situation, where the two tails are very tall, and a low distribution around the mean. The orderNorm transformation can help (see [Best Normalizator] section)

Interesting Predictors

Histograms

cols = c('x11','x18','stat98','x7','stat110')
df=gather(select_at(data,cols))
ggplot(df, aes(value)) + 
  geom_histogram(aes(y=..density..),bins = 50,fill='light blue') + 
  geom_density() + 
  # stat_function(fun = dnorm, n = 100, args = list(mean = mean(df$value), sd = sd(df$value)))  
  facet_wrap(~key, scales = 'free',ncol=3)

# ggplot(gather(select_at(data,cols)), aes(sample=value)) + 
#   stat_qq()+
#   facet_wrap(~key, scales = 'free',ncol=2)

lapply(select_at(data,cols),summary)

## $x11
##      Min.   1st Qu.    Median      Mean   3rd Qu.      Max. 
## 9.000e-08 9.494e-08 1.001e-07 1.001e-07 1.052e-07 1.100e-07 
## 
## $x18
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##   1.500   3.147   4.769   4.772   6.418   7.999 
## 
## $stat98
##      Min.   1st Qu.    Median      Mean   3rd Qu.      Max. 
## -2.998619 -1.551882 -0.015993 -0.005946  1.528405  2.999499 
## 
## $x7
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##   0.700   1.266   1.854   1.852   2.446   3.000 
## 
## $stat110
##      Min.   1st Qu.    Median      Mean   3rd Qu.      Max. 
## -2.999543 -1.496865 -0.002193 -0.004129  1.504273  2.999563

Scatter plot vs. output variable **y3.log

d = gather(dplyr::select_at(data,c(cols,output.var.tr)),key=target,value=value,-!!output.var.tr)
ggplot(data=d, aes_string(x='value',y=output.var.tr)) + 
  geom_point(color='light green',alpha=0.5) + 
  geom_smooth() + 
  facet_wrap(~target, scales = 'free',ncol=3)

## `geom_smooth()` using method = 'gam' and formula 'y ~ s(x, bs = "cs")'

All Predictors

Histograms

All indicators have a strong indication of Fat-Tails

df=gather(select_at(data,predictors))
ggplot(df, aes(value)) + 
  geom_histogram(aes(y=..density..),bins = 50,fill='light blue') + 
  geom_density() + 
  # stat_function(fun = dnorm, n = 100, args = list(mean = mean(df$value), sd = sd(df$value)))  
  facet_wrap(~key, scales = 'free',ncol=4)

Correlations

With Output Variable

#chart.Correlation(select(data,-JobName),  pch=21)
t=as.data.frame(round(cor(dplyr::select(data,-one_of(output.var.tr,'JobName'))
                          ,select_at(data,output.var.tr)),4))  %>%
  rownames_to_column(var='variable') %>% filter(variable != !!output.var) %>% arrange(-y3.log)
#DT::datatable(t)
message("Top Positive")

## Top Positive

kable(head(arrange(t,desc(y3.log)),20))

variable	y3.log
x18	0.3120
x7	0.2091
stat98	0.1784
x9	0.1127
x17	0.0611
x16	0.0489
x10	0.0472
x21	0.0412
x11	0.0322
x8	0.0318
stat156	0.0287
stat23	0.0234
stat100	0.0206
stat144	0.0203
stat59	0.0202
stat60	0.0199
stat195	0.0199
stat141	0.0194
stat73	0.0192
stat197	0.0185

message("Top Negative")

## Top Negative

kable(head(arrange(t,y3.log),20))

variable	y3.log
stat110	-0.1594
x4	-0.0603
stat13	-0.0345
stat41	-0.0345
stat14	-0.0317
stat149	-0.0309
stat113	-0.0279
stat4	-0.0248
stat106	-0.0236
stat146	-0.0236
stat186	-0.0217
stat91	-0.0210
stat214	-0.0209
stat5	-0.0207
stat22	-0.0202
stat39	-0.0202
stat175	-0.0194
stat187	-0.0193
stat128	-0.0192
stat37	-0.0191

Between All Variables

#chart.Correlation(select(data,-JobName),  pch=21)
t=as.data.frame(round(cor(dplyr::select(data,-one_of('JobName'))),4))
#DT::datatable(t,options=list(scrollX=T))
message("Showing only 10 variables")

## Showing only 10 variables

kable(t[1:10,1:10])

	x1	x2	x3	x4	x5	x6	x7	x8	x9	x10
x1	1.0000	0.0034	-0.0028	0.0085	0.0068	0.0159	0.0264	-0.0012	0.0142	0.0013
x2	0.0034	1.0000	-0.0057	0.0004	-0.0094	-0.0101	0.0089	0.0078	0.0049	-0.0214
x3	-0.0028	-0.0057	1.0000	0.0029	0.0046	0.0006	-0.0105	-0.0002	0.0167	-0.0137
x4	0.0085	0.0004	0.0029	1.0000	-0.0059	0.0104	0.0098	0.0053	0.0061	-0.0023
x5	0.0068	-0.0094	0.0046	-0.0059	1.0000	0.0016	-0.0027	0.0081	0.0259	-0.0081
x6	0.0159	-0.0101	0.0006	0.0104	0.0016	1.0000	0.0200	-0.0157	0.0117	-0.0072
x7	0.0264	0.0089	-0.0105	0.0098	-0.0027	0.0200	1.0000	-0.0018	-0.0069	-0.0221
x8	-0.0012	0.0078	-0.0002	0.0053	0.0081	-0.0157	-0.0018	1.0000	0.0142	-0.0004
x9	0.0142	0.0049	0.0167	0.0061	0.0259	0.0117	-0.0069	0.0142	1.0000	0.0149
x10	0.0013	-0.0214	-0.0137	-0.0023	-0.0081	-0.0072	-0.0221	-0.0004	0.0149	1.0000

Scatter Plots with Output Variable

Scatter plots with all predictors and the output variable (y3.log)

d = gather(dplyr::select_at(data,c(predictors,output.var.tr)),key=target,value=value,-!!output.var.tr)
ggplot(data=d, aes_string(x='value',y=output.var.tr)) + 
  geom_point(color='light blue',alpha=0.5) + 
  geom_smooth() + 
  facet_wrap(~target, scales = 'free',ncol=4)

## `geom_smooth()` using method = 'gam' and formula 'y ~ s(x, bs = "cs")'

Multicollinearity - VIF

No Multicollinearity among predictors

Showing Top predictor by VIF Value

vifDF = usdm::vif(select_at(data,predictors)) %>% arrange(desc(VIF))
head(vifDF,15)

##    Variables      VIF
## 1    stat200 1.064425
## 2    stat105 1.062772
## 3    stat129 1.060113
## 4        x22 1.059883
## 5    stat186 1.059724
## 6      stat2 1.059350
## 7     stat38 1.059264
## 8    stat124 1.059115
## 9     stat52 1.058905
## 10    stat72 1.058726
## 11       x10 1.058718
## 12    stat46 1.058439
## 13    stat32 1.058189
## 14   stat163 1.058128
## 15    stat20 1.057925

Feature Eng

Square Root transformation for x18

data.tr=data %>%
  mutate(x18.sqrt = sqrt(x18)) 
cols=c('x18','x18.sqrt')

Comparing Pre and Post Transformation Density Plots

# ggplot(gather(select_at(data.tr,cols)), aes(value)) + 
#   geom_histogram(aes(y=..density..),bins = 50,fill='light blue') + 
#   geom_density() + 
#   facet_wrap(~key, scales = 'free',ncol=4)

d = gather(dplyr::select_at(data.tr,c(cols,output.var.tr)),key=target,value=value,-!!output.var.tr)
ggplot(data=d, aes_string(x='value',y=output.var.tr)) + 
  geom_point(color='light blue',alpha=0.5) + 
  geom_smooth() + 
  facet_wrap(~target, scales = 'free',ncol=4)

## `geom_smooth()` using method = 'gam' and formula 'y ~ s(x, bs = "cs")'

#removing unwanted variables
data.tr=data.tr %>%
  dplyr::select_at(names(data.tr)[! names(data.tr) %in% c('x18','y3','JobName')])

data=data.tr
label.names=output.var.tr

Modeling

Train Test Split

data = data[sample(nrow(data)),] # randomly shuffle data
split = sample.split(data[,label.names], SplitRatio = 0.8)

data.train = subset(data, split == TRUE)
data.test = subset(data, split == FALSE)

Common Functions

plot.diagnostics <-  function(model, train) {
  plot(model)
  
  residuals = resid(model) # Plotted above in plot(lm.out)
  r.standard = rstandard(model)
  r.student = rstudent(model)
  
  df = data.frame(x=predict(model,train),y=r.student)
  p=ggplot(data=df,aes(x=x,y=y)) +
    geom_point(color='blue',alpha=0.5,shape=20,size=2) +
    geom_hline(yintercept = c(-2,0,2),size=1)+
    ylab("Student Residuals") +
    xlab("Predicted Values")+
    ggtitle("Standardized Residual Plot")
  plot(p)
  
  # df = data.frame(x=predict(model,train),y=r.standard)
  # p=ggplot(data=df,aes(x=x,y=y)) +
  #   geom_point(color='blue',alpha=0.5,shape=20,size=2) +
  #   geom_hline(yintercept = c(-2,0,2),size=1)+
  #   ylab("Standardized Residuals") +
  #   xlab("Predicted Values")+
  #   ggtitle("Student Residual Plot")
  # plot(p)
  # Histogram
  df=data.frame(r.student)
  p=ggplot(data=df,aes(r.student)) +
    geom_histogram(aes(y=..density..),bins = 50,fill='blue',alpha=0.6) + 
    stat_function(fun = dnorm, n = 100, args = list(mean = 0, sd = 1)) +
    ylab("Density")+
    xlab("Studentized Residuals")+
    ggtitle("Distribution of Studentized Residuals")
  plot(p)
  # http://www.stat.columbia.edu/~martin/W2024/R7.pdf
  # Influential plots
  inf.meas = influence.measures(model)
  # print (summary(inf.meas)) # too much data
  
  # Leverage plot
  lev = hat(model.matrix(model))
  df=tibble::rownames_to_column(as.data.frame(lev),'id')
  p=ggplot(data=df,aes(x=as.numeric(id),y=lev)) +
    geom_point(color='blue',alpha=0.5,shape=20,size=2) +
    ylab('Leverage - check') + 
    xlab('Index')
  plot(p)
  # Cook's Distance
  cd = cooks.distance(model)
  df=tibble::rownames_to_column(as.data.frame(cd),'id')
  p=ggplot(data=df,aes(x=as.numeric(id),y=cd)) +
    geom_point(color='blue',alpha=0.5,shape=20,size=2) +
    geom_text(data=filter(df,cd>15/nrow(train)),aes(label=id),check_overlap=T,size=3,vjust=-.5)+
    ylab('Cooks distances') + 
    geom_hline(yintercept = c(4/nrow(train),0),size=1)+
    xlab('Index')
  plot(p)
  print (paste("Number of data points that have Cook's D > 4/n: ", length(cd[cd > 4/nrow(train)]), sep = "")) 
  print (paste("Number of data points that have Cook's D > 1: ", length(cd[cd > 1]), sep = "")) 
  return(cd)
}

# function to set up random seeds
# Based on http://jaehyeon-kim.github.io/2015/05/Setup-Random-Seeds-on-Caret-Package.html 
setCaretSeeds <- function(method = "cv", numbers = 1, repeats = 1, tunes = NULL, seed = 1701) {
  #B is the number of resamples and integer vector of M (numbers + tune length if any)
  B <- if (method == "cv") numbers
  else if(method == "repeatedcv") numbers * repeats
  else NULL
  if(is.null(length)) {
    seeds <- NULL
  } else {
    set.seed(seed = seed)
    seeds <- vector(mode = "list", length = B)
    seeds <- lapply(seeds, function(x) sample.int(n = 1000000
                                                  , size = numbers + ifelse(is.null(tunes), 0, tunes)))
    seeds[[length(seeds) + 1]] <- sample.int(n = 1000000, size = 1)
  }
  # return seeds
  seeds
}

train.caret.glmselect = function(formula, data, method
                                 ,subopt = NULL, feature.names
                                 , train.control = NULL, tune.grid = NULL, pre.proc = NULL){
  
  if(is.null(train.control)){
    train.control <- trainControl(method = "cv"
                              ,number = 10
                              ,seeds = setCaretSeeds(method = "cv"
                                                     , numbers = 10
                                                     , seed = 1701)
                              ,search = "grid"
                              ,verboseIter = TRUE
                              ,allowParallel = TRUE
                              )
  }
  
  if(is.null(tune.grid)){
    if (method == 'leapForward' | method == 'leapBackward' | method == 'leapSeq'){
      tune.grid = data.frame(nvmax = 1:length(feature.names))
    }
    if (method == 'glmnet' && subopt == 'LASSO'){
      # Will only show 1 Lambda value during training, but that is OK
      # https://stackoverflow.com/questions/47526544/why-need-to-tune-lambda-with-carettrain-method-glmnet-and-cv-glmnet
      # Another option for LASSO is this: https://github.com/topepo/caret/blob/master/RegressionTests/Code/lasso.R
      lambda = 10^seq(-2,0, length =100)
      alpha = c(1)
      tune.grid = expand.grid(alpha = alpha,lambda = lambda)
    }
    if (method == 'lars'){
      # https://github.com/topepo/caret/blob/master/RegressionTests/Code/lars.R
      fraction = seq(0, 1, length = 100)
      tune.grid = expand.grid(fraction = fraction)
      pre.proc = c("center", "scale") 
    }
  }
  
  # http://sshaikh.org/2015/05/06/parallelize-machine-learning-in-r-with-multi-core-cpus/
  cl <- makeCluster(ceiling(detectCores()*0.85)) # use 75% of cores only, leave rest for other tasks
  registerDoParallel(cl)

  set.seed(1) 
  # note that the seed has to actually be set just before this function is called
  # settign is above just not ensure reproducibility for some reason
  model.caret <- caret::train(formula
                              , data = data
                              , method = method
                              , tuneGrid = tune.grid
                              , trControl = train.control
                              , preProc = pre.proc
                              )
  
  stopCluster(cl)
  registerDoSEQ() # register sequential engine in case you are not using this function anymore
  
  if (method == 'leapForward' | method == 'leapBackward' | method == 'leapSeq'){
    print("All models results")
    print(model.caret$results) # all model results
    print("Best Model")
    print(model.caret$bestTune) # best model
    model = model.caret$finalModel

    # Metrics Plot 
    dataPlot = model.caret$results %>%
      gather(key='metric',value='value',-nvmax) %>%
      dplyr::filter(metric %in% c('MAE','RMSE','Rsquared'))
    metricsPlot = ggplot(data=dataPlot,aes(x=nvmax,y=value) ) +
      geom_line(color='lightblue4') +
      geom_point(color='blue',alpha=0.7,size=.9) +
      facet_wrap(~metric,ncol=2,scales='free')+
      theme_light()
    plot(metricsPlot)
    
    # Residuals Plot
    # leap function does not support studentized residuals
    dataPlot=data.frame(pred=predict(model.caret,data),res=resid(model.caret))
    residPlot = ggplot(dataPlot,aes(x=pred,y=res)) +
      geom_point(color='light blue',alpha=0.7) +
      geom_smooth(method="lm")+
      theme_light()
    plot(residPlot)
   
    residHistogram = ggplot(dataPlot,aes(x=res)) +
      geom_histogram(aes(y=..density..),fill='light blue',alpha=1) +
      #geom_density(color='lightblue4') + 
      stat_function(fun = dnorm, n = 100, args = list(mean = mean(dataPlot$res)
                                                       , sd = sd(dataPlot$res)),color='lightblue4')  
      theme_light()
    plot(residHistogram)
    id = rownames(model.caret$bestTune)    
    # Provides the coefficients of the best model
    # regsubsets doens return a full model (see documentation of regsubset), so we need to recalcualte themodel
    # https://stackoverflow.com/questions/13063762/how-to-obtain-a-lm-object-from-regsubsets
    print("Coefficients of final model:")
    coefs <- coef(model, id=id)
    #calculate the model to the the coef intervals
    nams <- names(coefs)
    nams <- nams[!nams %in% "(Intercept)"]
    response <-  as.character(formula[[2]])
    form <- as.formula(paste(response, paste(nams, collapse = " + "), sep = " ~ "))
    mod <- lm(form, data = data)
    #coefs
    #coef(mod)
    print(car::Confint(mod))
    return(list(model = model,id = id, residPlot = residPlot, residHistogram=residHistogram
                ,modelLM=mod))
  }
  if (method == 'glmnet' && subopt == 'LASSO'){
    print(model.caret)
    print(plot(model.caret))
    print(model.caret$bestTune)
    
    print(model.caret$results)
    model=model.caret$finalModel
    # Metrics Plot 
    dataPlot = model.caret$results %>%
      gather(key='metric',value='value',-lambda) %>%
      dplyr::filter(metric %in% c('MAE','RMSE','Rsquared'))
    metricsPlot = ggplot(data=dataPlot,aes(x=lambda,y=value) ) +
      geom_line(color='lightblue4') +
      geom_point(color='blue',alpha=0.7,size=.9) +
      facet_wrap(~metric,ncol=2,scales='free')+
      theme_light()
    plot(metricsPlot)
    
    # Residuals Plot 
    dataPlot=data.frame(pred=predict(model.caret,data),res=resid(model.caret))
    residPlot = ggplot(dataPlot,aes(x=pred,y=res)) +
      geom_point(color='light blue',alpha=0.7) +
      geom_smooth(method="lm")+
      theme_light()
    plot(residPlot)

    residHistogram = ggplot(dataPlot,aes(x=res)) +
      geom_histogram(aes(y=..density..),fill='light blue',alpha=1) +
      #geom_density(color='lightblue4') +
      stat_function(fun = dnorm, n = 100, args = list(mean = mean(dataPlot$res)
                                                       , sd = sd(dataPlot$res)),color='lightblue4')  
      theme_light()
    plot(residHistogram)
    
    print("Coefficients") 
    #no interval for glmnet: https://stackoverflow.com/questions/39750965/confidence-intervals-for-ridge-regression
    t=coef(model,s=model.caret$bestTune$lambda)
    model.coef = t[which(t[,1]!=0),]
    print(as.data.frame(model.coef))
    id = NULL # not really needed but added for consistency
    return(list(model = model.caret,id = id, residPlot = residPlot, metricsPlot=metricsPlot ))
  }
  if (method == 'lars'){
    print(model.caret)
    print(plot(model.caret))
    print(model.caret$bestTune)
    
    # Metrics Plot
    dataPlot = model.caret$results %>%
        gather(key='metric',value='value',-fraction) %>%
      dplyr::filter(metric %in% c('MAE','RMSE','Rsquared'))
    metricsPlot = ggplot(data=dataPlot,aes(x=fraction,y=value) ) +
      geom_line(color='lightblue4') +
      geom_point(color='blue',alpha=0.7,size=.9) +
      facet_wrap(~metric,ncol=2,scales='free')+
      theme_light()
    plot(metricsPlot)
    
    # Residuals Plot
    dataPlot=data.frame(pred=predict(model.caret,data),res=resid(model.caret))
    residPlot = ggplot(dataPlot,aes(x=pred,y=res)) +
      geom_point(color='light blue',alpha=0.7) +
      geom_smooth(method="lm")+
      theme_light()
    plot(residPlot)

    residHistogram = ggplot(dataPlot,aes(x=res)) +
      geom_histogram(aes(y=..density..),fill='light blue',alpha=1) +
      #geom_density(color='lightblue4') + 
      stat_function(fun = dnorm, n = 100, args = list(mean = mean(dataPlot$res)
                                                       , sd = sd(dataPlot$res)),color='lightblue4')  
      theme_light()
    plot(residHistogram)
    
    print("Coefficients") 
    t=coef(model.caret$finalModel,s=model.caret$bestTune$fraction,mode='fraction')
    model.coef = t[which(t!=0)]
    print(model.coef)
    id = NULL # not really needed but added for consistency
    return(list(model = model.caret,id = id, residPlot = residPlot, residHistogram=residHistogram))
  }
}

# https://stackoverflow.com/questions/48265743/linear-model-subset-selection-goodness-of-fit-with-k-fold-cross-validation
# changed slightly since call[[2]] was just returning "formula" without actually returnign the value in formula
predict.regsubsets <- function(object, newdata, id, formula, ...) {
    #form <- as.formula(object$call[[2]])
    mat <- model.matrix(formula, newdata) # adds intercept and expands any interaction terms
    coefi <- coef(object, id = id)
    xvars <- names(coefi)
    return(mat[,xvars]%*%coefi)
}
  
test.model = function(model, test, level=0.95
                      ,draw.limits = FALSE, good = 0.1, ok = 0.15
                      ,method = NULL, subopt = NULL
                      ,id = NULL, formula, feature.names, label.names
                      ,transformation = NULL){
  ## if using caret for glm select equivalent functionality, 
  ## need to pass formula (full is ok as it will select subset of variables from there)
  if (is.null(method)){
    pred = predict(model, newdata=test, interval="confidence", level = level) 
  }
  
  if (method == 'leapForward' | method == 'leapBackward' | method == 'leapSeq'){
    pred = predict.regsubsets(model, newdata = test, id = id, formula = formula)
  }
  
  if (method == 'glmnet' && subopt == 'LASSO'){
    xtest = as.matrix(test[,feature.names]) 
    pred=as.data.frame(predict(model, xtest))
  }
  
  if (method == 'lars'){
    pred=as.data.frame(predict(model, newdata = test))
  }
    
  # Summary of predicted values
  print ("Summary of predicted values: ")
  print(summary(pred[,1]))

  test.mse = mean((test[,label.names]-pred[,1])^2)
  print (paste(method, subopt, "Test MSE:", test.mse, sep=" "))
  
  if(log.pred == TRUE || norm.pred == TRUE){
    # plot transformewd comparison first
    df=data.frame(x=test[,label.names],y=pred[,1])
    ggplot(df,aes(x=x,y=y)) +
      geom_point(color='blue',alpha=0.5,shape=20,size=2) +
      geom_abline(slope=1,intercept=0,color='black',size=1) +
      #scale_y_continuous(limits=c(min(df),max(df)))+
      xlab("Actual (Transformed)")+
      ylab("Predicted (Transformed)")
  }
    
  if (log.pred == FALSE && norm.pred == FALSE){
    x = test[,label.names]
    y = pred[,1]
  }
  if (log.pred == TRUE){
    x = 10^test[,label.names]
    y = 10^pred[,1]  
  }
  if (norm.pred == TRUE){
    x = predict(transformation, test[,label.names], inverse = TRUE)
    y = predict(transformation, pred[,1], inverse = TRUE)
  }

  df=data.frame(x,y)
  ggplot(df,aes(x,y)) +
    geom_point(color='blue',alpha=0.5,shape=20,size=2) +
    geom_abline(slope=c(1+good,1-good,1+ok,1-ok)
                ,intercept=rep(0,4),color=c('dark green','dark green','dark red','dark red'),size=1,alpha=0.8) +
    #scale_y_continuous(limits=c(min(df),max(df)))+
    xlab("Actual")+
    ylab("Predicted") 
    
 
}

Setup Formulae

n <- names(data.train)
 formula <- as.formula(paste(paste(n[n %in% label.names], collapse = " + ")
                             ," ~", paste(n[!n %in% label.names], collapse = " + "))) 

grand.mean.formula = as.formula(paste(paste(n[n %in% label.names], collapse = " + ")," ~ 1"))

print(formula)

## y3.log ~ x1 + x2 + x3 + x4 + x5 + x6 + x7 + x8 + x9 + x10 + x11 + 
##     x12 + x13 + x14 + x15 + x16 + x17 + x19 + x20 + x21 + x22 + 
##     x23 + stat1 + stat2 + stat3 + stat4 + stat5 + stat6 + stat7 + 
##     stat8 + stat9 + stat10 + stat11 + stat12 + stat13 + stat14 + 
##     stat15 + stat16 + stat17 + stat18 + stat19 + stat20 + stat21 + 
##     stat22 + stat23 + stat24 + stat25 + stat26 + stat27 + stat28 + 
##     stat29 + stat30 + stat31 + stat32 + stat33 + stat34 + stat35 + 
##     stat36 + stat37 + stat38 + stat39 + stat40 + stat41 + stat42 + 
##     stat43 + stat44 + stat45 + stat46 + stat47 + stat48 + stat49 + 
##     stat50 + stat51 + stat52 + stat53 + stat54 + stat55 + stat56 + 
##     stat57 + stat58 + stat59 + stat60 + stat61 + stat62 + stat63 + 
##     stat64 + stat65 + stat66 + stat67 + stat68 + stat69 + stat70 + 
##     stat71 + stat72 + stat73 + stat74 + stat75 + stat76 + stat77 + 
##     stat78 + stat79 + stat80 + stat81 + stat82 + stat83 + stat84 + 
##     stat85 + stat86 + stat87 + stat88 + stat89 + stat90 + stat91 + 
##     stat92 + stat93 + stat94 + stat95 + stat96 + stat97 + stat98 + 
##     stat99 + stat100 + stat101 + stat102 + stat103 + stat104 + 
##     stat105 + stat106 + stat107 + stat108 + stat109 + stat110 + 
##     stat111 + stat112 + stat113 + stat114 + stat115 + stat116 + 
##     stat117 + stat118 + stat119 + stat120 + stat121 + stat122 + 
##     stat123 + stat124 + stat125 + stat126 + stat127 + stat128 + 
##     stat129 + stat130 + stat131 + stat132 + stat133 + stat134 + 
##     stat135 + stat136 + stat137 + stat138 + stat139 + stat140 + 
##     stat141 + stat142 + stat143 + stat144 + stat145 + stat146 + 
##     stat147 + stat148 + stat149 + stat150 + stat151 + stat152 + 
##     stat153 + stat154 + stat155 + stat156 + stat157 + stat158 + 
##     stat159 + stat160 + stat161 + stat162 + stat163 + stat164 + 
##     stat165 + stat166 + stat167 + stat168 + stat169 + stat170 + 
##     stat171 + stat172 + stat173 + stat174 + stat175 + stat176 + 
##     stat177 + stat178 + stat179 + stat180 + stat181 + stat182 + 
##     stat183 + stat184 + stat185 + stat186 + stat187 + stat188 + 
##     stat189 + stat190 + stat191 + stat192 + stat193 + stat194 + 
##     stat195 + stat196 + stat197 + stat198 + stat199 + stat200 + 
##     stat201 + stat202 + stat203 + stat204 + stat205 + stat206 + 
##     stat207 + stat208 + stat209 + stat210 + stat211 + stat212 + 
##     stat213 + stat214 + stat215 + stat216 + stat217 + x18.sqrt

print(grand.mean.formula)

## y3.log ~ 1

# Update feature.names because we may have transformed some features
feature.names = n[!n %in% label.names]

Full Model

model.full = lm(formula , data.train)
summary(model.full)

## 
## Call:
## lm(formula = formula, data = data.train)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -0.08178 -0.02067 -0.00471  0.01609  0.18639 
## 
## Coefficients:
##               Estimate Std. Error t value Pr(>|t|)    
## (Intercept)  1.970e+00  9.556e-03 206.142  < 2e-16 ***
## x1          -6.863e-05  6.498e-04  -0.106 0.915881    
## x2           5.008e-05  4.179e-04   0.120 0.904605    
## x3           1.337e-04  1.138e-04   1.175 0.240228    
## x4          -5.131e-05  9.020e-06  -5.688 1.35e-08 ***
## x5           2.699e-04  2.947e-04   0.916 0.359764    
## x6          -2.488e-04  5.985e-04  -0.416 0.677570    
## x7           1.130e-02  6.383e-04  17.709  < 2e-16 ***
## x8           4.326e-04  1.477e-04   2.928 0.003421 ** 
## x9           3.098e-03  3.315e-04   9.344  < 2e-16 ***
## x10          1.321e-03  3.063e-04   4.313 1.64e-05 ***
## x11          1.603e+05  7.391e+04   2.169 0.030132 *  
## x12         -1.902e-04  1.878e-04  -1.013 0.311326    
## x13          3.608e-05  7.489e-05   0.482 0.630021    
## x14         -1.027e-04  3.226e-04  -0.318 0.750218    
## x15         -2.033e-05  3.076e-04  -0.066 0.947299    
## x16          9.357e-04  2.143e-04   4.366 1.29e-05 ***
## x17          1.654e-03  3.244e-04   5.100 3.52e-07 ***
## x19          3.010e-04  1.653e-04   1.821 0.068680 .  
## x20          1.207e-04  1.147e-03   0.105 0.916227    
## x21          1.313e-04  4.213e-05   3.115 0.001847 ** 
## x22         -4.289e-04  3.438e-04  -1.248 0.212236    
## x23         -2.949e-04  3.260e-04  -0.905 0.365689    
## stat1       -2.484e-04  2.503e-04  -0.992 0.321105    
## stat2        1.218e-05  2.468e-04   0.049 0.960647    
## stat3        1.824e-04  2.487e-04   0.733 0.463312    
## stat4       -5.521e-04  2.484e-04  -2.222 0.026296 *  
## stat5       -2.208e-04  2.482e-04  -0.889 0.373802    
## stat6       -1.547e-04  2.478e-04  -0.624 0.532420    
## stat7       -1.437e-04  2.487e-04  -0.578 0.563516    
## stat8        3.158e-04  2.485e-04   1.271 0.203879    
## stat9       -8.143e-05  2.468e-04  -0.330 0.741472    
## stat10      -3.091e-04  2.479e-04  -1.247 0.212454    
## stat11      -2.275e-04  2.492e-04  -0.913 0.361337    
## stat12      -5.211e-05  2.478e-04  -0.210 0.833444    
## stat13      -3.516e-04  2.460e-04  -1.429 0.152977    
## stat14      -9.129e-04  2.478e-04  -3.684 0.000232 ***
## stat15      -4.564e-04  2.467e-04  -1.850 0.064417 .  
## stat16       1.833e-04  2.476e-04   0.740 0.459108    
## stat17      -2.116e-05  2.447e-04  -0.086 0.931105    
## stat18      -1.787e-04  2.470e-04  -0.724 0.469329    
## stat19      -9.858e-05  2.472e-04  -0.399 0.690112    
## stat20      -3.307e-04  2.480e-04  -1.334 0.182388    
## stat21      -1.325e-04  2.486e-04  -0.533 0.594097    
## stat22      -5.268e-04  2.468e-04  -2.134 0.032886 *  
## stat23       7.187e-04  2.465e-04   2.916 0.003564 ** 
## stat24      -4.782e-04  2.483e-04  -1.926 0.054202 .  
## stat25      -5.482e-04  2.461e-04  -2.227 0.025973 *  
## stat26      -2.560e-04  2.456e-04  -1.043 0.297167    
## stat27       7.786e-05  2.487e-04   0.313 0.754268    
## stat28      -9.652e-05  2.473e-04  -0.390 0.696361    
## stat29       1.491e-04  2.493e-04   0.598 0.549822    
## stat30       3.020e-04  2.512e-04   1.202 0.229229    
## stat31      -6.386e-05  2.515e-04  -0.254 0.799563    
## stat32       1.056e-04  2.508e-04   0.421 0.673699    
## stat33      -2.304e-04  2.480e-04  -0.929 0.352959    
## stat34       1.075e-04  2.468e-04   0.435 0.663256    
## stat35      -5.684e-04  2.470e-04  -2.301 0.021423 *  
## stat36      -9.813e-05  2.478e-04  -0.396 0.692087    
## stat37      -5.752e-04  2.517e-04  -2.285 0.022333 *  
## stat38       5.418e-04  2.494e-04   2.173 0.029836 *  
## stat39      -2.344e-04  2.460e-04  -0.953 0.340839    
## stat40       4.738e-05  2.482e-04   0.191 0.848628    
## stat41      -6.205e-04  2.463e-04  -2.519 0.011804 *  
## stat42      -3.248e-04  2.479e-04  -1.310 0.190087    
## stat43      -2.760e-04  2.500e-04  -1.104 0.269573    
## stat44       1.183e-04  2.483e-04   0.477 0.633699    
## stat45      -3.276e-04  2.475e-04  -1.324 0.185697    
## stat46       3.546e-04  2.473e-04   1.434 0.151635    
## stat47      -2.842e-05  2.486e-04  -0.114 0.908990    
## stat48       2.542e-04  2.477e-04   1.026 0.304777    
## stat49       2.000e-04  2.463e-04   0.812 0.416818    
## stat50       7.916e-05  2.456e-04   0.322 0.747192    
## stat51       2.703e-04  2.473e-04   1.093 0.274348    
## stat52      -2.712e-04  2.478e-04  -1.095 0.273746    
## stat53      -2.500e-04  2.497e-04  -1.001 0.316923    
## stat54      -4.110e-04  2.500e-04  -1.644 0.100192    
## stat55       1.406e-04  2.447e-04   0.575 0.565631    
## stat56      -2.634e-04  2.476e-04  -1.064 0.287411    
## stat57      -3.385e-06  2.449e-04  -0.014 0.988972    
## stat58      -4.144e-05  2.468e-04  -0.168 0.866656    
## stat59       3.593e-04  2.476e-04   1.451 0.146848    
## stat60       5.944e-04  2.487e-04   2.390 0.016866 *  
## stat61      -1.471e-04  2.486e-04  -0.592 0.554073    
## stat62      -8.530e-05  2.474e-04  -0.345 0.730298    
## stat63       1.823e-04  2.481e-04   0.735 0.462328    
## stat64       5.864e-05  2.469e-04   0.238 0.812246    
## stat65      -3.736e-04  2.489e-04  -1.501 0.133326    
## stat66       2.772e-04  2.535e-04   1.093 0.274334    
## stat67       6.872e-06  2.492e-04   0.028 0.978006    
## stat68       1.337e-04  2.466e-04   0.542 0.587730    
## stat69       1.408e-04  2.482e-04   0.567 0.570508    
## stat70       1.925e-04  2.466e-04   0.780 0.435164    
## stat71      -3.814e-05  2.452e-04  -0.156 0.876362    
## stat72       3.268e-04  2.492e-04   1.311 0.189769    
## stat73       1.849e-04  2.478e-04   0.746 0.455710    
## stat74       6.229e-05  2.483e-04   0.251 0.801932    
## stat75      -6.380e-05  2.489e-04  -0.256 0.797678    
## stat76       3.682e-05  2.470e-04   0.149 0.881517    
## stat77      -8.797e-05  2.465e-04  -0.357 0.721246    
## stat78      -1.196e-04  2.482e-04  -0.482 0.629851    
## stat79      -1.945e-04  2.487e-04  -0.782 0.434157    
## stat80       2.584e-04  2.480e-04   1.042 0.297565    
## stat81       2.283e-04  2.508e-04   0.911 0.362595    
## stat82       3.252e-04  2.469e-04   1.317 0.187740    
## stat83      -8.618e-05  2.473e-04  -0.348 0.727517    
## stat84      -2.504e-04  2.469e-04  -1.014 0.310449    
## stat85       2.483e-04  2.492e-04   0.996 0.319100    
## stat86       4.428e-05  2.481e-04   0.178 0.858365    
## stat87       4.049e-05  2.488e-04   0.163 0.870741    
## stat88      -9.589e-05  2.453e-04  -0.391 0.695839    
## stat89      -1.515e-04  2.456e-04  -0.617 0.537468    
## stat90      -1.667e-04  2.493e-04  -0.669 0.503690    
## stat91      -3.512e-04  2.462e-04  -1.427 0.153703    
## stat92      -3.541e-04  2.471e-04  -1.433 0.151945    
## stat93      -6.548e-05  2.499e-04  -0.262 0.793338    
## stat94      -8.931e-05  2.483e-04  -0.360 0.719055    
## stat95      -1.625e-04  2.474e-04  -0.657 0.511194    
## stat96      -4.889e-04  2.468e-04  -1.981 0.047607 *  
## stat97       1.114e-04  2.460e-04   0.453 0.650746    
## stat98       3.297e-03  2.426e-04  13.593  < 2e-16 ***
## stat99       3.826e-04  2.486e-04   1.539 0.123847    
## stat100      5.573e-04  2.480e-04   2.247 0.024649 *  
## stat101      2.894e-07  2.501e-04   0.001 0.999077    
## stat102      1.036e-04  2.499e-04   0.414 0.678677    
## stat103     -5.811e-04  2.519e-04  -2.307 0.021074 *  
## stat104     -1.892e-04  2.477e-04  -0.764 0.444832    
## stat105      1.596e-04  2.455e-04   0.650 0.515621    
## stat106     -3.576e-04  2.475e-04  -1.445 0.148491    
## stat107      8.019e-05  2.480e-04   0.323 0.746472    
## stat108     -1.348e-04  2.476e-04  -0.545 0.586090    
## stat109     -6.745e-05  2.478e-04  -0.272 0.785510    
## stat110     -3.227e-03  2.464e-04 -13.093  < 2e-16 ***
## stat111     -7.776e-05  2.460e-04  -0.316 0.751965    
## stat112     -2.546e-05  2.499e-04  -0.102 0.918851    
## stat113     -3.882e-04  2.499e-04  -1.553 0.120424    
## stat114      6.827e-05  2.479e-04   0.275 0.783051    
## stat115      2.100e-04  2.472e-04   0.850 0.395559    
## stat116      2.002e-04  2.506e-04   0.799 0.424324    
## stat117      1.081e-04  2.491e-04   0.434 0.664400    
## stat118     -5.108e-04  2.460e-04  -2.076 0.037927 *  
## stat119      2.218e-04  2.483e-04   0.893 0.371742    
## stat120      6.950e-05  2.469e-04   0.281 0.778392    
## stat121     -2.689e-04  2.478e-04  -1.085 0.277837    
## stat122     -1.433e-04  2.463e-04  -0.582 0.560863    
## stat123     -2.120e-05  2.522e-04  -0.084 0.933010    
## stat124     -1.940e-04  2.477e-04  -0.783 0.433590    
## stat125      3.736e-05  2.468e-04   0.151 0.879677    
## stat126      2.449e-04  2.459e-04   0.996 0.319196    
## stat127      2.319e-05  2.471e-04   0.094 0.925230    
## stat128     -1.690e-04  2.462e-04  -0.687 0.492365    
## stat129      1.295e-04  2.462e-04   0.526 0.599026    
## stat130      2.281e-04  2.496e-04   0.914 0.360818    
## stat131      9.077e-05  2.480e-04   0.366 0.714413    
## stat132      1.151e-04  2.464e-04   0.467 0.640561    
## stat133      1.998e-04  2.477e-04   0.806 0.420069    
## stat134     -2.182e-04  2.464e-04  -0.885 0.375959    
## stat135     -2.970e-05  2.471e-04  -0.120 0.904321    
## stat136      1.480e-05  2.488e-04   0.059 0.952573    
## stat137      1.172e-04  2.454e-04   0.478 0.632959    
## stat138     -1.551e-04  2.469e-04  -0.628 0.529876    
## stat139      3.655e-06  2.500e-04   0.015 0.988334    
## stat140      3.697e-05  2.471e-04   0.150 0.881076    
## stat141      2.219e-04  2.465e-04   0.900 0.368063    
## stat142     -8.024e-05  2.503e-04  -0.321 0.748552    
## stat143      2.102e-04  2.473e-04   0.850 0.395429    
## stat144      6.758e-04  2.473e-04   2.732 0.006312 ** 
## stat145     -8.687e-05  2.506e-04  -0.347 0.728862    
## stat146     -3.701e-04  2.498e-04  -1.482 0.138526    
## stat147     -3.864e-04  2.495e-04  -1.548 0.121607    
## stat148     -4.145e-04  2.443e-04  -1.697 0.089830 .  
## stat149     -4.791e-04  2.500e-04  -1.916 0.055395 .  
## stat150      8.583e-05  2.491e-04   0.345 0.730440    
## stat151     -8.858e-05  2.499e-04  -0.354 0.723043    
## stat152     -2.637e-04  2.473e-04  -1.066 0.286304    
## stat153      3.258e-05  2.516e-04   0.129 0.896982    
## stat154      1.469e-04  2.498e-04   0.588 0.556458    
## stat155     -2.144e-04  2.462e-04  -0.871 0.383821    
## stat156      5.697e-04  2.514e-04   2.266 0.023465 *  
## stat157     -1.609e-04  2.458e-04  -0.654 0.512862    
## stat158     -2.509e-05  2.514e-04  -0.100 0.920482    
## stat159      2.146e-04  2.470e-04   0.869 0.385085    
## stat160      8.115e-06  2.480e-04   0.033 0.973894    
## stat161      1.319e-04  2.486e-04   0.531 0.595758    
## stat162      1.029e-04  2.458e-04   0.419 0.675345    
## stat163     -5.019e-05  2.513e-04  -0.200 0.841666    
## stat164      3.362e-04  2.496e-04   1.347 0.178027    
## stat165      2.758e-05  2.469e-04   0.112 0.911029    
## stat166     -2.798e-04  2.454e-04  -1.140 0.254237    
## stat167     -1.416e-04  2.471e-04  -0.573 0.566662    
## stat168     -3.821e-04  2.476e-04  -1.543 0.122821    
## stat169     -3.081e-05  2.486e-04  -0.124 0.901383    
## stat170     -1.655e-04  2.481e-04  -0.667 0.504870    
## stat171      6.706e-06  2.498e-04   0.027 0.978585    
## stat172      1.666e-04  2.475e-04   0.673 0.500939    
## stat173     -2.884e-04  2.492e-04  -1.157 0.247305    
## stat174     -5.387e-05  2.488e-04  -0.217 0.828605    
## stat175     -2.137e-04  2.482e-04  -0.861 0.389264    
## stat176      8.150e-05  2.472e-04   0.330 0.741654    
## stat177     -1.097e-04  2.496e-04  -0.439 0.660339    
## stat178      1.420e-04  2.508e-04   0.566 0.571409    
## stat179      1.283e-04  2.480e-04   0.517 0.604879    
## stat180     -1.137e-04  2.460e-04  -0.462 0.643946    
## stat181      1.680e-04  2.492e-04   0.674 0.500346    
## stat182      7.516e-05  2.487e-04   0.302 0.762486    
## stat183      4.747e-06  2.485e-04   0.019 0.984763    
## stat184     -5.071e-05  2.477e-04  -0.205 0.837816    
## stat185     -1.533e-04  2.437e-04  -0.629 0.529253    
## stat186     -1.041e-04  2.498e-04  -0.417 0.676742    
## stat187     -3.417e-04  2.476e-04  -1.380 0.167680    
## stat188     -2.035e-04  2.480e-04  -0.821 0.411894    
## stat189      2.199e-04  2.485e-04   0.885 0.376140    
## stat190      8.906e-05  2.473e-04   0.360 0.718794    
## stat191     -1.377e-04  2.476e-04  -0.556 0.578200    
## stat192     -1.280e-04  2.501e-04  -0.512 0.609013    
## stat193      3.807e-05  2.510e-04   0.152 0.879449    
## stat194      2.531e-04  2.472e-04   1.024 0.305926    
## stat195      3.924e-04  2.485e-04   1.579 0.114453    
## stat196     -2.738e-04  2.509e-04  -1.091 0.275144    
## stat197     -3.489e-05  2.446e-04  -0.143 0.886591    
## stat198     -4.807e-04  2.486e-04  -1.934 0.053183 .  
## stat199     -2.719e-05  2.445e-04  -0.111 0.911451    
## stat200     -7.682e-05  2.437e-04  -0.315 0.752575    
## stat201     -1.626e-04  2.476e-04  -0.657 0.511296    
## stat202     -2.217e-04  2.499e-04  -0.887 0.375097    
## stat203     -8.868e-05  2.463e-04  -0.360 0.718780    
## stat204     -6.357e-04  2.455e-04  -2.589 0.009648 ** 
## stat205     -4.788e-05  2.456e-04  -0.195 0.845406    
## stat206     -7.061e-06  2.496e-04  -0.028 0.977429    
## stat207      3.499e-04  2.486e-04   1.407 0.159456    
## stat208      2.750e-04  2.489e-04   1.105 0.269234    
## stat209      1.437e-04  2.461e-04   0.584 0.559344    
## stat210     -8.051e-05  2.498e-04  -0.322 0.747214    
## stat211     -2.218e-04  2.491e-04  -0.891 0.373216    
## stat212      1.354e-04  2.483e-04   0.545 0.585596    
## stat213     -1.129e-04  2.490e-04  -0.453 0.650404    
## stat214     -2.589e-04  2.468e-04  -1.049 0.294243    
## stat215     -2.433e-04  2.473e-04  -0.984 0.325180    
## stat216     -1.774e-05  2.483e-04  -0.071 0.943042    
## stat217     -8.501e-05  2.479e-04  -0.343 0.731615    
## x18.sqrt     2.653e-02  9.419e-04  28.165  < 2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.03137 on 5343 degrees of freedom
## Multiple R-squared:  0.2666, Adjusted R-squared:  0.2336 
## F-statistic: 8.092 on 240 and 5343 DF,  p-value: < 2.2e-16

cd.full = plot.diagnostics(model=model.full, train=data.train)

## [1] "Number of data points that have Cook's D > 4/n: 283"
## [1] "Number of data points that have Cook's D > 1: 0"

Checking with removal of high influence points

high.cd = names(cd.full[cd.full > 4/nrow(data.train)])
data.train2 = data.train[!(rownames(data.train)) %in% high.cd,]
model.full2 = lm(formula , data.train2)
summary(model.full2)

## 
## Call:
## lm(formula = formula, data = data.train2)
## 
## Residuals:
##       Min        1Q    Median        3Q       Max 
## -0.059672 -0.017473 -0.002468  0.016181  0.070186 
## 
## Coefficients:
##               Estimate Std. Error t value Pr(>|t|)    
## (Intercept)  1.954e+00  7.844e-03 249.112  < 2e-16 ***
## x1          -1.006e-04  5.332e-04  -0.189 0.850396    
## x2           2.787e-04  3.418e-04   0.816 0.414784    
## x3           6.475e-05  9.285e-05   0.697 0.485587    
## x4          -5.863e-05  7.393e-06  -7.930 2.68e-15 ***
## x5           3.638e-04  2.407e-04   1.512 0.130721    
## x6          -4.620e-04  4.900e-04  -0.943 0.345768    
## x7           1.225e-02  5.228e-04  23.429  < 2e-16 ***
## x8           5.193e-04  1.209e-04   4.296 1.77e-05 ***
## x9           3.034e-03  2.707e-04  11.207  < 2e-16 ***
## x10          1.667e-03  2.510e-04   6.641 3.44e-11 ***
## x11          2.274e+05  6.071e+04   3.745 0.000182 ***
## x12         -2.858e-05  1.534e-04  -0.186 0.852152    
## x13          1.054e-04  6.150e-05   1.713 0.086739 .  
## x14          2.813e-05  2.637e-04   0.107 0.915041    
## x15          9.537e-07  2.519e-04   0.004 0.996980    
## x16          9.754e-04  1.753e-04   5.566 2.75e-08 ***
## x17          1.773e-03  2.658e-04   6.670 2.82e-11 ***
## x19          2.625e-04  1.354e-04   1.938 0.052682 .  
## x20          7.106e-04  9.405e-04   0.756 0.449935    
## x21          1.404e-04  3.447e-05   4.073 4.71e-05 ***
## x22         -6.247e-04  2.810e-04  -2.223 0.026258 *  
## x23          1.560e-05  2.671e-04   0.058 0.953417    
## stat1       -2.928e-04  2.047e-04  -1.430 0.152736    
## stat2       -2.099e-05  2.021e-04  -0.104 0.917290    
## stat3        2.453e-04  2.034e-04   1.206 0.227908    
## stat4       -7.082e-04  2.040e-04  -3.472 0.000520 ***
## stat5       -2.878e-04  2.037e-04  -1.413 0.157727    
## stat6       -2.288e-04  2.027e-04  -1.129 0.259040    
## stat7       -2.218e-04  2.031e-04  -1.092 0.274801    
## stat8        1.441e-04  2.031e-04   0.710 0.477996    
## stat9       -2.105e-04  2.022e-04  -1.041 0.297965    
## stat10      -3.519e-04  2.025e-04  -1.738 0.082270 .  
## stat11      -2.708e-04  2.037e-04  -1.329 0.183788    
## stat12      -1.873e-04  2.026e-04  -0.924 0.355354    
## stat13      -4.118e-04  2.014e-04  -2.044 0.040957 *  
## stat14      -1.067e-03  2.026e-04  -5.267 1.44e-07 ***
## stat15      -7.284e-04  2.020e-04  -3.606 0.000314 ***
## stat16       1.821e-05  2.025e-04   0.090 0.928336    
## stat17      -1.715e-04  2.005e-04  -0.856 0.392173    
## stat18      -6.926e-05  2.020e-04  -0.343 0.731736    
## stat19      -2.102e-05  2.029e-04  -0.104 0.917487    
## stat20       8.344e-05  2.028e-04   0.411 0.680725    
## stat21      -7.701e-05  2.035e-04  -0.378 0.705146    
## stat22      -3.973e-04  2.016e-04  -1.971 0.048771 *  
## stat23       5.283e-04  2.022e-04   2.612 0.009020 ** 
## stat24      -5.404e-04  2.035e-04  -2.655 0.007948 ** 
## stat25      -3.525e-04  2.014e-04  -1.750 0.080146 .  
## stat26      -4.003e-04  2.013e-04  -1.989 0.046747 *  
## stat27      -8.281e-05  2.042e-04  -0.406 0.685111    
## stat28      -1.719e-04  2.025e-04  -0.849 0.395936    
## stat29       1.848e-04  2.041e-04   0.905 0.365284    
## stat30       1.995e-04  2.052e-04   0.972 0.330931    
## stat31      -2.877e-05  2.061e-04  -0.140 0.888980    
## stat32       9.292e-05  2.056e-04   0.452 0.651341    
## stat33      -2.685e-04  2.030e-04  -1.322 0.186062    
## stat34       2.723e-04  2.018e-04   1.349 0.177305    
## stat35      -6.636e-04  2.024e-04  -3.279 0.001049 ** 
## stat36      -4.160e-05  2.030e-04  -0.205 0.837637    
## stat37      -3.116e-04  2.063e-04  -1.511 0.130929    
## stat38       6.079e-04  2.039e-04   2.982 0.002880 ** 
## stat39      -2.945e-04  2.007e-04  -1.467 0.142330    
## stat40       8.267e-05  2.035e-04   0.406 0.684545    
## stat41      -6.399e-04  2.012e-04  -3.181 0.001477 ** 
## stat42      -1.812e-04  2.029e-04  -0.893 0.371769    
## stat43      -1.650e-04  2.042e-04  -0.808 0.419022    
## stat44       1.170e-04  2.033e-04   0.576 0.564962    
## stat45      -2.082e-04  2.026e-04  -1.028 0.304145    
## stat46       3.088e-04  2.027e-04   1.524 0.127661    
## stat47       2.265e-04  2.030e-04   1.116 0.264541    
## stat48       2.709e-04  2.023e-04   1.339 0.180592    
## stat49      -4.536e-05  2.022e-04  -0.224 0.822481    
## stat50       1.033e-04  2.008e-04   0.515 0.606793    
## stat51       1.737e-05  2.026e-04   0.086 0.931685    
## stat52      -6.611e-05  2.027e-04  -0.326 0.744379    
## stat53      -1.627e-04  2.044e-04  -0.796 0.426073    
## stat54      -3.784e-04  2.051e-04  -1.845 0.065100 .  
## stat55      -2.782e-05  2.002e-04  -0.139 0.889491    
## stat56       2.477e-05  2.025e-04   0.122 0.902630    
## stat57      -1.329e-04  2.009e-04  -0.662 0.508283    
## stat58       1.851e-05  2.013e-04   0.092 0.926738    
## stat59       3.648e-04  2.025e-04   1.801 0.071687 .  
## stat60       5.623e-04  2.032e-04   2.767 0.005675 ** 
## stat61      -2.213e-04  2.033e-04  -1.089 0.276268    
## stat62      -1.958e-04  2.023e-04  -0.968 0.332973    
## stat63       1.454e-04  2.037e-04   0.714 0.475190    
## stat64       1.307e-04  2.018e-04   0.647 0.517357    
## stat65      -2.407e-04  2.036e-04  -1.182 0.237162    
## stat66       2.188e-04  2.070e-04   1.057 0.290736    
## stat67      -1.933e-05  2.036e-04  -0.095 0.924378    
## stat68      -5.693e-05  2.017e-04  -0.282 0.777806    
## stat69       1.349e-04  2.032e-04   0.664 0.506619    
## stat70       1.750e-04  2.019e-04   0.867 0.386055    
## stat71       6.304e-05  2.012e-04   0.313 0.754034    
## stat72       1.778e-04  2.041e-04   0.871 0.383742    
## stat73       2.600e-04  2.030e-04   1.281 0.200268    
## stat74       2.486e-04  2.030e-04   1.225 0.220716    
## stat75       1.303e-04  2.037e-04   0.640 0.522230    
## stat76      -4.703e-06  2.020e-04  -0.023 0.981432    
## stat77       1.447e-04  2.017e-04   0.717 0.473367    
## stat78      -3.502e-04  2.028e-04  -1.727 0.084185 .  
## stat79       1.699e-04  2.030e-04   0.837 0.402707    
## stat80       2.814e-04  2.028e-04   1.388 0.165342    
## stat81       8.358e-05  2.055e-04   0.407 0.684175    
## stat82       1.103e-04  2.021e-04   0.546 0.585214    
## stat83      -1.369e-04  2.026e-04  -0.676 0.499038    
## stat84      -3.122e-04  2.017e-04  -1.548 0.121672    
## stat85      -1.715e-04  2.041e-04  -0.840 0.400723    
## stat86       2.374e-05  2.028e-04   0.117 0.906815    
## stat87       2.507e-05  2.035e-04   0.123 0.901973    
## stat88       1.306e-04  2.008e-04   0.651 0.515372    
## stat89       1.152e-04  2.013e-04   0.572 0.567113    
## stat90      -2.045e-04  2.041e-04  -1.002 0.316299    
## stat91      -3.246e-04  2.009e-04  -1.615 0.106277    
## stat92      -4.085e-04  2.021e-04  -2.021 0.043346 *  
## stat93       7.728e-05  2.055e-04   0.376 0.706891    
## stat94      -2.198e-05  2.028e-04  -0.108 0.913711    
## stat95       1.737e-04  2.031e-04   0.855 0.392589    
## stat96      -5.336e-04  2.021e-04  -2.641 0.008300 ** 
## stat97       2.497e-04  2.009e-04   1.243 0.214083    
## stat98       3.355e-03  1.982e-04  16.929  < 2e-16 ***
## stat99       3.620e-04  2.034e-04   1.780 0.075083 .  
## stat100      5.890e-04  2.027e-04   2.906 0.003674 ** 
## stat101      9.050e-05  2.046e-04   0.442 0.658287    
## stat102      1.746e-04  2.045e-04   0.854 0.393229    
## stat103     -6.227e-04  2.061e-04  -3.021 0.002529 ** 
## stat104     -1.356e-04  2.032e-04  -0.667 0.504605    
## stat105      2.101e-04  2.007e-04   1.046 0.295413    
## stat106     -3.913e-04  2.023e-04  -1.934 0.053149 .  
## stat107      3.594e-05  2.033e-04   0.177 0.859640    
## stat108     -5.122e-05  2.026e-04  -0.253 0.800410    
## stat109     -1.632e-04  2.031e-04  -0.803 0.421796    
## stat110     -3.159e-03  2.013e-04 -15.689  < 2e-16 ***
## stat111     -3.348e-05  2.010e-04  -0.167 0.867692    
## stat112     -2.269e-05  2.046e-04  -0.111 0.911705    
## stat113     -2.960e-04  2.044e-04  -1.448 0.147620    
## stat114      6.672e-05  2.034e-04   0.328 0.742962    
## stat115      3.671e-04  2.025e-04   1.813 0.069944 .  
## stat116      2.890e-04  2.049e-04   1.410 0.158471    
## stat117      9.480e-05  2.035e-04   0.466 0.641280    
## stat118     -2.474e-04  2.015e-04  -1.228 0.219682    
## stat119      3.059e-04  2.028e-04   1.508 0.131528    
## stat120     -6.368e-05  2.019e-04  -0.315 0.752507    
## stat121     -3.261e-04  2.027e-04  -1.609 0.107675    
## stat122     -1.608e-04  2.019e-04  -0.796 0.425861    
## stat123      2.330e-04  2.060e-04   1.131 0.258192    
## stat124     -1.939e-04  2.025e-04  -0.957 0.338448    
## stat125     -9.369e-05  2.024e-04  -0.463 0.643373    
## stat126      1.675e-04  2.013e-04   0.832 0.405190    
## stat127     -6.926e-05  2.017e-04  -0.343 0.731342    
## stat128     -3.811e-04  2.009e-04  -1.897 0.057830 .  
## stat129      4.311e-05  2.013e-04   0.214 0.830418    
## stat130      2.485e-04  2.041e-04   1.218 0.223269    
## stat131      1.917e-05  2.028e-04   0.095 0.924694    
## stat132      3.908e-05  2.016e-04   0.194 0.846334    
## stat133      3.193e-04  2.030e-04   1.573 0.115805    
## stat134     -1.567e-04  2.014e-04  -0.778 0.436538    
## stat135     -1.540e-04  2.023e-04  -0.761 0.446515    
## stat136     -5.277e-05  2.035e-04  -0.259 0.795427    
## stat137      9.876e-05  2.007e-04   0.492 0.622760    
## stat138     -1.760e-04  2.024e-04  -0.870 0.384555    
## stat139     -7.530e-05  2.045e-04  -0.368 0.712721    
## stat140      9.920e-05  2.015e-04   0.492 0.622478    
## stat141      2.497e-04  2.017e-04   1.238 0.215866    
## stat142     -8.143e-05  2.046e-04  -0.398 0.690714    
## stat143      5.200e-05  2.023e-04   0.257 0.797158    
## stat144      7.088e-04  2.023e-04   3.504 0.000462 ***
## stat145     -2.182e-04  2.054e-04  -1.062 0.288062    
## stat146     -6.001e-04  2.043e-04  -2.937 0.003330 ** 
## stat147     -3.444e-04  2.045e-04  -1.684 0.092190 .  
## stat148     -3.123e-04  2.003e-04  -1.559 0.119059    
## stat149     -3.351e-04  2.045e-04  -1.638 0.101394    
## stat150     -1.043e-04  2.042e-04  -0.511 0.609465    
## stat151      2.195e-04  2.053e-04   1.069 0.285008    
## stat152     -2.542e-04  2.023e-04  -1.257 0.208852    
## stat153      2.024e-04  2.055e-04   0.985 0.324770    
## stat154      2.432e-04  2.046e-04   1.189 0.234686    
## stat155     -1.390e-04  2.016e-04  -0.690 0.490454    
## stat156      5.234e-04  2.053e-04   2.550 0.010810 *  
## stat157     -1.696e-04  2.006e-04  -0.845 0.397966    
## stat158      2.330e-04  2.055e-04   1.134 0.256993    
## stat159      3.911e-04  2.017e-04   1.939 0.052558 .  
## stat160     -2.707e-05  2.037e-04  -0.133 0.894290    
## stat161      5.193e-05  2.035e-04   0.255 0.798547    
## stat162      1.120e-04  2.007e-04   0.558 0.576784    
## stat163      1.314e-04  2.063e-04   0.637 0.524066    
## stat164      2.301e-04  2.046e-04   1.125 0.260648    
## stat165      1.607e-04  2.025e-04   0.793 0.427602    
## stat166     -2.457e-04  2.005e-04  -1.226 0.220441    
## stat167     -2.124e-04  2.020e-04  -1.052 0.293055    
## stat168     -2.128e-04  2.024e-04  -1.052 0.293015    
## stat169     -2.002e-04  2.041e-04  -0.981 0.326531    
## stat170      1.649e-05  2.030e-04   0.081 0.935260    
## stat171     -1.599e-04  2.045e-04  -0.782 0.434133    
## stat172      4.611e-04  2.020e-04   2.283 0.022451 *  
## stat173     -1.003e-04  2.037e-04  -0.492 0.622492    
## stat174      1.386e-04  2.037e-04   0.680 0.496394    
## stat175     -8.449e-05  2.029e-04  -0.416 0.677102    
## stat176     -1.987e-04  2.022e-04  -0.983 0.325867    
## stat177     -3.254e-04  2.041e-04  -1.594 0.110895    
## stat178      1.071e-04  2.049e-04   0.523 0.601182    
## stat179      6.874e-05  2.030e-04   0.339 0.734939    
## stat180     -5.548e-05  2.018e-04  -0.275 0.783403    
## stat181      1.769e-04  2.040e-04   0.867 0.385816    
## stat182      1.996e-04  2.038e-04   0.979 0.327485    
## stat183      2.331e-05  2.037e-04   0.114 0.908904    
## stat184      1.267e-04  2.025e-04   0.626 0.531589    
## stat185     -7.414e-05  1.995e-04  -0.372 0.710118    
## stat186      2.143e-04  2.040e-04   1.051 0.293506    
## stat187     -1.355e-04  2.025e-04  -0.669 0.503313    
## stat188     -4.129e-05  2.030e-04  -0.203 0.838822    
## stat189     -9.430e-05  2.037e-04  -0.463 0.643459    
## stat190     -7.275e-05  2.027e-04  -0.359 0.719646    
## stat191     -1.645e-04  2.029e-04  -0.811 0.417479    
## stat192     -1.536e-05  2.048e-04  -0.075 0.940204    
## stat193      1.027e-04  2.056e-04   0.500 0.617238    
## stat194      1.954e-04  2.027e-04   0.964 0.335126    
## stat195      2.507e-05  2.035e-04   0.123 0.901973    
## stat196     -2.533e-04  2.053e-04  -1.234 0.217238    
## stat197     -6.852e-05  2.006e-04  -0.342 0.732648    
## stat198     -4.724e-04  2.032e-04  -2.325 0.020087 *  
## stat199     -2.222e-05  1.999e-04  -0.111 0.911528    
## stat200     -2.620e-05  2.000e-04  -0.131 0.895812    
## stat201     -7.635e-05  2.028e-04  -0.376 0.706604    
## stat202     -6.241e-05  2.045e-04  -0.305 0.760246    
## stat203     -2.131e-06  2.016e-04  -0.011 0.991566    
## stat204     -4.278e-04  2.012e-04  -2.126 0.033537 *  
## stat205      2.614e-04  2.005e-04   1.304 0.192308    
## stat206      2.265e-05  2.043e-04   0.111 0.911720    
## stat207      3.200e-04  2.037e-04   1.571 0.116291    
## stat208      2.732e-04  2.037e-04   1.341 0.179968    
## stat209      4.938e-05  2.009e-04   0.246 0.805818    
## stat210     -3.287e-04  2.042e-04  -1.609 0.107633    
## stat211     -1.374e-04  2.037e-04  -0.674 0.500032    
## stat212      1.592e-04  2.032e-04   0.784 0.433367    
## stat213     -6.417e-05  2.033e-04  -0.316 0.752321    
## stat214     -5.880e-05  2.022e-04  -0.291 0.771161    
## stat215     -3.556e-05  2.025e-04  -0.176 0.860585    
## stat216     -4.683e-05  2.030e-04  -0.231 0.817546    
## stat217     -6.784e-05  2.026e-04  -0.335 0.737703    
## x18.sqrt     2.573e-02  7.695e-04  33.442  < 2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.02498 on 5060 degrees of freedom
## Multiple R-squared:  0.3716, Adjusted R-squared:  0.3418 
## F-statistic: 12.47 on 240 and 5060 DF,  p-value: < 2.2e-16

cd.full2 = plot.diagnostics(model.full2, data.train2)

## [1] "Number of data points that have Cook's D > 4/n: 268"
## [1] "Number of data points that have Cook's D > 1: 0"

# much more normal residuals than before. 
# Checking to see if distributions are different and if so whcih variables
# High Leverage Plot 
plotData = data.train %>% 
  rownames_to_column() %>%
  mutate(type=ifelse(rowname %in% high.cd,'High','Normal')) %>%
  dplyr::select(type,target=one_of(label.names))

ggplot(data=plotData, aes(x=type,y=target)) +
  geom_boxplot(fill='light blue',outlier.shape=NA) +
  scale_y_continuous(name="Target Variable Values",label=scales::comma_format(accuracy=.1)) +
  theme_light() +
  ggtitle('Distribution of High Leverage Points and Normal  Points')

# 2 sample t-tests

plotData = data.train %>% 
  rownames_to_column() %>%
  mutate(type=ifelse(rowname %in% high.cd,'High','Normal')) %>%
  dplyr::select(type,one_of(feature.names))

comp.test = lapply(dplyr::select(plotData, one_of(feature.names))
                   , function(x) t.test(x ~ plotData$type, var.equal = TRUE)) 

sig.comp = list.filter(comp.test, p.value < 0.05)
sapply(sig.comp, function(x) x[['p.value']])

##          x4         x13      stat47      stat74      stat79      stat85      stat95      stat98     stat110     stat118 
## 0.016745266 0.048060296 0.010332348 0.022937363 0.023191401 0.014578179 0.012976838 0.004766528 0.004564928 0.049750199 
##     stat128     stat146     stat151     stat172     stat174    x18.sqrt 
## 0.020931056 0.002432513 0.002668840 0.015474014 0.016114109 0.005719575

mm = melt(plotData, id=c('type')) %>% filter(variable %in% names(sig.comp))

ggplot(mm,aes(x=type, y=value)) +
  geom_boxplot()+
  facet_wrap(~variable, ncol=5, scales = 'free_y') +
  scale_y_continuous(name="values",label=scales::comma_format(accuracy=.1)) +
  ggtitle('Distribution of High Leverage Points and Normal Points')

# Distribution (box) Plots
mm = melt(plotData, id=c('type'))

ggplot(mm,aes(x=type, y=value)) +
  geom_boxplot()+
  facet_wrap(~variable, ncol=8, scales = 'free_y') +
  scale_y_continuous(name="values",label=scales::comma_format(accuracy=.1)) +
  ggtitle('Distribution of High Leverage Points and Normal Points')

Grand Means Model

model.null = lm(grand.mean.formula, data.train)
summary(model.null)

## 
## Call:
## lm(formula = grand.mean.formula, data = data.train)
## 
## Residuals:
##       Min        1Q    Median        3Q       Max 
## -0.114447 -0.023670 -0.003088  0.020699  0.190865 
## 
## Coefficients:
##              Estimate Std. Error t value Pr(>|t|)    
## (Intercept) 2.0963232  0.0004796    4371   <2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.03584 on 5583 degrees of freedom

Variable Selection

Basic: http://www.stat.columbia.edu/~martin/W2024/R10.pdf Cross Validation + Other Metrics: http://www.sthda.com/english/articles/37-model-selection-essentials-in-r/154-stepwise-regression-essentials-in-r/

Forward Selection with CV

Train

if (algo.forward.caret == TRUE){
  set.seed(1)
  returned = train.caret.glmselect(formula = formula
                                   , data = data.train
                                   , method = "leapForward"
                                   , feature.names = feature.names)
  model.forward = returned$model
  id = returned$id
}

## Aggregating results
## Selecting tuning parameters
## Fitting nvmax = 9 on full training set
## [1] "All models results"
##     nvmax       RMSE  Rsquared        MAE      RMSESD RsquaredSD        MAESD
## 1       1 0.03376443 0.1132051 0.02637850 0.001210125 0.02571127 0.0007953495
## 2       2 0.03297241 0.1543836 0.02566825 0.001331813 0.02849647 0.0008195492
## 3       3 0.03250196 0.1779147 0.02515184 0.001317023 0.02808594 0.0007621952
## 4       4 0.03196246 0.2052676 0.02446499 0.001452681 0.03363383 0.0008687720
## 5       5 0.03171086 0.2179498 0.02429750 0.001512147 0.03600988 0.0008518664
## 6       6 0.03165522 0.2206821 0.02425311 0.001516391 0.03578302 0.0008635422
## 7       7 0.03161930 0.2224386 0.02422076 0.001591134 0.03757869 0.0009392440
## 8       8 0.03155244 0.2257309 0.02419734 0.001617014 0.03868599 0.0009624076
## 9       9 0.03146134 0.2301155 0.02411499 0.001613656 0.03869431 0.0009595924
## 10     10 0.03148846 0.2287059 0.02413610 0.001591183 0.03762129 0.0009509402
## 11     11 0.03149049 0.2285003 0.02414479 0.001570434 0.03691604 0.0009354630
## 12     12 0.03150288 0.2280520 0.02416044 0.001590991 0.03717482 0.0009408043
## 13     13 0.03147622 0.2293254 0.02413800 0.001586666 0.03717877 0.0009728358
## 14     14 0.03148550 0.2289291 0.02413604 0.001600811 0.03767695 0.0009991203
## 15     15 0.03151049 0.2277667 0.02415137 0.001606362 0.03779119 0.0009983448
## 16     16 0.03151635 0.2273578 0.02416525 0.001572932 0.03632568 0.0009714183
## 17     17 0.03150438 0.2281246 0.02415982 0.001625619 0.03860212 0.0010191859
## 18     18 0.03148935 0.2287951 0.02414977 0.001611874 0.03811350 0.0009983158
## 19     19 0.03151416 0.2275893 0.02417693 0.001615300 0.03794160 0.0009956441
## 20     20 0.03152056 0.2272807 0.02417600 0.001596570 0.03716561 0.0009803289
## 21     21 0.03154669 0.2261143 0.02419830 0.001617907 0.03751442 0.0009939951
## 22     22 0.03154258 0.2263547 0.02419706 0.001629603 0.03753844 0.0009951548
## 23     23 0.03153604 0.2267287 0.02419708 0.001655219 0.03857136 0.0010094637
## 24     24 0.03155255 0.2260822 0.02419265 0.001684882 0.03957765 0.0010268588
## 25     25 0.03155408 0.2260284 0.02418587 0.001689191 0.03990833 0.0010270298
## 26     26 0.03154873 0.2262955 0.02416219 0.001685026 0.03986879 0.0010210127
## 27     27 0.03155328 0.2260303 0.02416425 0.001663973 0.03967169 0.0010144567
## 28     28 0.03155382 0.2260225 0.02416367 0.001663828 0.03962532 0.0010069188
## 29     29 0.03156583 0.2254547 0.02418043 0.001651293 0.03897893 0.0009943649
## 30     30 0.03155684 0.2258888 0.02415646 0.001638154 0.03841906 0.0009837442
## 31     31 0.03153999 0.2266618 0.02414505 0.001632233 0.03819485 0.0009886043
## 32     32 0.03151698 0.2277475 0.02412685 0.001637437 0.03866256 0.0009879513
## 33     33 0.03151831 0.2276883 0.02413576 0.001645905 0.03911899 0.0009916034
## 34     34 0.03151883 0.2276606 0.02413974 0.001631469 0.03880302 0.0009744988
## 35     35 0.03154507 0.2264281 0.02416116 0.001629429 0.03864976 0.0009707139
## 36     36 0.03156000 0.2257663 0.02417609 0.001637191 0.03888266 0.0009651906
## 37     37 0.03157255 0.2252214 0.02417586 0.001629348 0.03887564 0.0009433069
## 38     38 0.03157709 0.2250000 0.02417403 0.001619733 0.03834612 0.0009490266
## 39     39 0.03158311 0.2246788 0.02418235 0.001608177 0.03811691 0.0009327292
## 40     40 0.03157660 0.2249812 0.02417328 0.001601715 0.03802827 0.0009333391
## 41     41 0.03157994 0.2248479 0.02417578 0.001598479 0.03767096 0.0009240815
## 42     42 0.03159280 0.2243115 0.02419645 0.001610375 0.03793595 0.0009412851
## 43     43 0.03161078 0.2235172 0.02420986 0.001619250 0.03824488 0.0009352708
## 44     44 0.03161571 0.2233960 0.02421582 0.001629535 0.03815850 0.0009466490
## 45     45 0.03163338 0.2225880 0.02423854 0.001624007 0.03781633 0.0009572754
## 46     46 0.03165357 0.2217409 0.02424698 0.001638198 0.03833703 0.0009517115
## 47     47 0.03165045 0.2218831 0.02425060 0.001641539 0.03855126 0.0009613354
## 48     48 0.03165229 0.2218124 0.02424900 0.001635124 0.03801085 0.0009475429
## 49     49 0.03165748 0.2215394 0.02426206 0.001617272 0.03745263 0.0009329003
## 50     50 0.03165576 0.2216631 0.02426824 0.001626231 0.03760716 0.0009422135
## 51     51 0.03165318 0.2218348 0.02425618 0.001646914 0.03847422 0.0009634940
## 52     52 0.03164534 0.2222132 0.02425384 0.001650135 0.03854186 0.0009693838
## 53     53 0.03164915 0.2220794 0.02425284 0.001655188 0.03864325 0.0009803038
## 54     54 0.03166404 0.2214448 0.02426685 0.001661344 0.03888532 0.0009801524
## 55     55 0.03167813 0.2208456 0.02428079 0.001654840 0.03868200 0.0009720576
## 56     56 0.03169254 0.2202632 0.02428812 0.001671027 0.03923630 0.0009800438
## 57     57 0.03169290 0.2202503 0.02428713 0.001660515 0.03880597 0.0009792344
## 58     58 0.03170811 0.2196494 0.02430801 0.001675331 0.03929542 0.0009813584
## 59     59 0.03170832 0.2196100 0.02430829 0.001679190 0.03949799 0.0009876783
## 60     60 0.03171442 0.2193722 0.02431397 0.001681803 0.03940761 0.0009880787
## 61     61 0.03171827 0.2192246 0.02431394 0.001682356 0.03954285 0.0009945707
## 62     62 0.03172933 0.2186956 0.02431744 0.001675110 0.03925966 0.0009916646
## 63     63 0.03173473 0.2184783 0.02431872 0.001676780 0.03925339 0.0009919249
## 64     64 0.03174445 0.2180452 0.02432738 0.001664626 0.03860308 0.0009848951
## 65     65 0.03174443 0.2180567 0.02433362 0.001667985 0.03876814 0.0009845751
## 66     66 0.03175913 0.2173833 0.02433572 0.001659284 0.03839698 0.0009783930
## 67     67 0.03178079 0.2164414 0.02435910 0.001664050 0.03827764 0.0009732358
## 68     68 0.03178756 0.2161682 0.02436393 0.001664038 0.03809367 0.0009690576
## 69     69 0.03178825 0.2161694 0.02436803 0.001659617 0.03798711 0.0009643715
## 70     70 0.03178399 0.2163411 0.02436652 0.001655424 0.03792377 0.0009597160
## 71     71 0.03179906 0.2156736 0.02437714 0.001647574 0.03757939 0.0009528316
## 72     72 0.03179416 0.2158477 0.02437399 0.001631353 0.03675672 0.0009342997
## 73     73 0.03179572 0.2157435 0.02437089 0.001612803 0.03599641 0.0009218926
## 74     74 0.03180787 0.2152322 0.02438424 0.001614303 0.03595790 0.0009241667
## 75     75 0.03181700 0.2148192 0.02439246 0.001610985 0.03605262 0.0009237233
## 76     76 0.03182615 0.2144628 0.02440048 0.001624772 0.03671604 0.0009362699
## 77     77 0.03183813 0.2139326 0.02440424 0.001625314 0.03655623 0.0009278254
## 78     78 0.03183733 0.2140381 0.02440092 0.001636449 0.03700436 0.0009362784
## 79     79 0.03184456 0.2137585 0.02439755 0.001630896 0.03670791 0.0009275353
## 80     80 0.03184634 0.2136726 0.02439934 0.001627336 0.03657316 0.0009292705
## 81     81 0.03184712 0.2136560 0.02440049 0.001632079 0.03670798 0.0009339735
## 82     82 0.03185246 0.2133743 0.02439837 0.001624579 0.03631407 0.0009321499
## 83     83 0.03185596 0.2132599 0.02439630 0.001615867 0.03578217 0.0009218415
## 84     84 0.03186888 0.2127285 0.02441174 0.001622706 0.03584910 0.0009243956
## 85     85 0.03187198 0.2125986 0.02441358 0.001622276 0.03585038 0.0009269831
## 86     86 0.03187488 0.2125116 0.02441085 0.001626148 0.03589743 0.0009260710
## 87     87 0.03188425 0.2120934 0.02442174 0.001623465 0.03567756 0.0009244922
## 88     88 0.03189377 0.2116576 0.02443020 0.001625477 0.03584241 0.0009306201
## 89     89 0.03189489 0.2115921 0.02442992 0.001611914 0.03543400 0.0009244797
## 90     90 0.03189803 0.2114880 0.02443294 0.001621994 0.03592825 0.0009350216
## 91     91 0.03189387 0.2116790 0.02443728 0.001624224 0.03617286 0.0009445513
## 92     92 0.03190258 0.2113163 0.02444987 0.001630018 0.03618382 0.0009481915
## 93     93 0.03191214 0.2109291 0.02446380 0.001625956 0.03589556 0.0009439625
## 94     94 0.03191953 0.2106278 0.02447116 0.001634376 0.03621845 0.0009504174
## 95     95 0.03192174 0.2105287 0.02447374 0.001639284 0.03646714 0.0009494909
## 96     96 0.03192383 0.2104574 0.02447592 0.001647418 0.03679894 0.0009502987
## 97     97 0.03192138 0.2105360 0.02447681 0.001645036 0.03661491 0.0009444669
## 98     98 0.03192558 0.2103441 0.02448593 0.001638947 0.03641185 0.0009402478
## 99     99 0.03193996 0.2097225 0.02450039 0.001634728 0.03603204 0.0009324637
## 100   100 0.03194713 0.2093916 0.02450688 0.001631410 0.03583493 0.0009282906
## 101   101 0.03195059 0.2092510 0.02450933 0.001633273 0.03589106 0.0009284388
## 102   102 0.03195079 0.2092721 0.02450903 0.001636954 0.03604166 0.0009276216
## 103   103 0.03195039 0.2092890 0.02451361 0.001634318 0.03600958 0.0009288547
## 104   104 0.03194778 0.2094480 0.02450635 0.001647475 0.03641458 0.0009352793
## 105   105 0.03195300 0.2092568 0.02450945 0.001653611 0.03653122 0.0009386915
## 106   106 0.03194676 0.2095200 0.02450036 0.001652690 0.03645639 0.0009413139
## 107   107 0.03194424 0.2096773 0.02450002 0.001657674 0.03680850 0.0009457516
## 108   108 0.03194396 0.2096988 0.02449708 0.001665709 0.03720448 0.0009561827
## 109   109 0.03194474 0.2097188 0.02450526 0.001674121 0.03755448 0.0009649439
## 110   110 0.03195409 0.2093471 0.02451377 0.001679231 0.03763577 0.0009648311
## 111   111 0.03195493 0.2093532 0.02450762 0.001680674 0.03768724 0.0009664962
## 112   112 0.03195505 0.2093891 0.02450460 0.001681926 0.03753329 0.0009662913
## 113   113 0.03195344 0.2095011 0.02451027 0.001682498 0.03754671 0.0009621606
## 114   114 0.03194959 0.2096658 0.02450635 0.001679427 0.03746315 0.0009609270
## 115   115 0.03195215 0.2095677 0.02450614 0.001681046 0.03754224 0.0009651858
## 116   116 0.03194839 0.2097654 0.02450252 0.001681647 0.03757296 0.0009651146
## 117   117 0.03195556 0.2094604 0.02451090 0.001681443 0.03752031 0.0009654871
## 118   118 0.03195898 0.2093186 0.02451724 0.001680721 0.03740338 0.0009643888
## 119   119 0.03196935 0.2088877 0.02452516 0.001679762 0.03727511 0.0009707635
## 120   120 0.03196945 0.2088651 0.02452932 0.001671420 0.03695710 0.0009620223
## 121   121 0.03197467 0.2086596 0.02453194 0.001676487 0.03718816 0.0009648268
## 122   122 0.03197439 0.2086852 0.02453554 0.001676644 0.03725109 0.0009658998
## 123   123 0.03198316 0.2083381 0.02453924 0.001679531 0.03743020 0.0009680243
## 124   124 0.03198606 0.2082206 0.02453967 0.001673898 0.03715531 0.0009599227
## 125   125 0.03198593 0.2082440 0.02453984 0.001676011 0.03722132 0.0009602820
## 126   126 0.03199250 0.2079521 0.02454583 0.001682366 0.03741507 0.0009644955
## 127   127 0.03199796 0.2077411 0.02454516 0.001691226 0.03772996 0.0009768143
## 128   128 0.03200385 0.2074923 0.02454675 0.001693681 0.03777845 0.0009797929
## 129   129 0.03200606 0.2074080 0.02454354 0.001693154 0.03785897 0.0009819511
## 130   130 0.03200100 0.2076367 0.02454008 0.001689598 0.03782988 0.0009808613
## 131   131 0.03200213 0.2076187 0.02454153 0.001696574 0.03808547 0.0009836103
## 132   132 0.03200088 0.2076828 0.02454062 0.001702009 0.03817866 0.0009870940
## 133   133 0.03200424 0.2075501 0.02454504 0.001701815 0.03820049 0.0009836945
## 134   134 0.03200675 0.2074742 0.02454620 0.001710244 0.03849383 0.0009903456
## 135   135 0.03201056 0.2073325 0.02455297 0.001717647 0.03875545 0.0009971011
## 136   136 0.03201873 0.2070125 0.02455705 0.001722382 0.03886395 0.0010013594
## 137   137 0.03201720 0.2070643 0.02455675 0.001719466 0.03890377 0.0009997024
## 138   138 0.03201959 0.2069799 0.02455816 0.001718972 0.03892620 0.0009972627
## 139   139 0.03202704 0.2066557 0.02456806 0.001719693 0.03896455 0.0009997890
## 140   140 0.03203111 0.2064998 0.02457047 0.001726596 0.03921937 0.0010020771
## 141   141 0.03203204 0.2064275 0.02456939 0.001723251 0.03921163 0.0010054752
## 142   142 0.03203737 0.2062058 0.02457312 0.001721927 0.03921771 0.0010009187
## 143   143 0.03203468 0.2063634 0.02457031 0.001730254 0.03955454 0.0010098625
## 144   144 0.03203194 0.2065047 0.02456799 0.001734497 0.03960300 0.0010138648
## 145   145 0.03202888 0.2066253 0.02456749 0.001732453 0.03949143 0.0010101781
## 146   146 0.03202900 0.2065828 0.02456817 0.001726149 0.03926183 0.0010086948
## 147   147 0.03203265 0.2064451 0.02456831 0.001728947 0.03934584 0.0010082405
## 148   148 0.03203265 0.2064716 0.02456818 0.001729942 0.03937349 0.0010105181
## 149   149 0.03203925 0.2062020 0.02457187 0.001729300 0.03941662 0.0010134382
## 150   150 0.03204270 0.2060345 0.02457140 0.001728513 0.03939170 0.0010124932
## 151   151 0.03204191 0.2060832 0.02457259 0.001732850 0.03951611 0.0010177971
## 152   152 0.03204591 0.2059142 0.02457287 0.001732439 0.03946691 0.0010154070
## 153   153 0.03204730 0.2058458 0.02457219 0.001732411 0.03941944 0.0010128432
## 154   154 0.03204789 0.2058192 0.02457299 0.001728224 0.03930249 0.0010124778
## 155   155 0.03204728 0.2058491 0.02457448 0.001729026 0.03937989 0.0010112018
## 156   156 0.03204863 0.2057925 0.02457229 0.001730239 0.03938558 0.0010079784
## 157   157 0.03205002 0.2057508 0.02457339 0.001733028 0.03944596 0.0010087673
## 158   158 0.03204928 0.2058014 0.02457474 0.001735593 0.03957032 0.0010092366
## 159   159 0.03205041 0.2057696 0.02457329 0.001737223 0.03966633 0.0010096001
## 160   160 0.03205335 0.2056615 0.02457434 0.001736527 0.03962676 0.0010107099
## 161   161 0.03205277 0.2056699 0.02457582 0.001733803 0.03952035 0.0010093485
## 162   162 0.03205657 0.2055293 0.02457856 0.001737473 0.03959675 0.0010106169
## 163   163 0.03205895 0.2054412 0.02458144 0.001742425 0.03980948 0.0010120838
## 164   164 0.03206061 0.2053590 0.02457867 0.001739526 0.03967356 0.0010072814
## 165   165 0.03205824 0.2054717 0.02457618 0.001740622 0.03973945 0.0010081460
## 166   166 0.03205788 0.2054671 0.02457549 0.001737660 0.03965766 0.0010084729
## 167   167 0.03205856 0.2054331 0.02457806 0.001734901 0.03963067 0.0010096999
## 168   168 0.03205826 0.2054423 0.02458130 0.001736207 0.03967999 0.0010107704
## 169   169 0.03205653 0.2055342 0.02458053 0.001738721 0.03978588 0.0010118609
## 170   170 0.03205692 0.2055109 0.02457902 0.001739632 0.03986380 0.0010096672
## 171   171 0.03205964 0.2054054 0.02457857 0.001741526 0.03999012 0.0010113802
## 172   172 0.03205775 0.2054849 0.02457659 0.001744494 0.04013054 0.0010120259
## 173   173 0.03206104 0.2053320 0.02457814 0.001742426 0.04004003 0.0010091491
## 174   174 0.03206154 0.2053388 0.02457995 0.001745361 0.04016413 0.0010114263
## 175   175 0.03206326 0.2052625 0.02458365 0.001748928 0.04030923 0.0010152442
## 176   176 0.03206449 0.2052126 0.02458324 0.001749031 0.04023667 0.0010146625
## 177   177 0.03206468 0.2051944 0.02458241 0.001745893 0.04014386 0.0010125679
## 178   178 0.03206690 0.2050950 0.02458418 0.001746059 0.04020591 0.0010140861
## 179   179 0.03206963 0.2049935 0.02458641 0.001744099 0.04012297 0.0010125982
## 180   180 0.03206824 0.2050605 0.02458589 0.001747495 0.04029491 0.0010137075
## 181   181 0.03206783 0.2050861 0.02458614 0.001748000 0.04032684 0.0010137868
## 182   182 0.03206601 0.2051526 0.02458461 0.001745317 0.04022556 0.0010134454
## 183   183 0.03206719 0.2051005 0.02458521 0.001740300 0.04005921 0.0010120811
## 184   184 0.03206570 0.2051739 0.02458287 0.001740030 0.04000019 0.0010109548
## 185   185 0.03206394 0.2052533 0.02458097 0.001739748 0.03994777 0.0010113347
## 186   186 0.03206590 0.2051779 0.02458136 0.001742400 0.04003026 0.0010123898
## 187   187 0.03206373 0.2052643 0.02457841 0.001739239 0.03995552 0.0010106592
## 188   188 0.03206450 0.2052473 0.02457848 0.001741928 0.04004459 0.0010128976
## 189   189 0.03206461 0.2052330 0.02457829 0.001742197 0.04008339 0.0010143514
## 190   190 0.03206515 0.2052026 0.02457903 0.001743699 0.04011418 0.0010144082
## 191   191 0.03206830 0.2050664 0.02458149 0.001743854 0.04008097 0.0010129111
## 192   192 0.03207039 0.2049842 0.02458347 0.001745013 0.04009942 0.0010152572
## 193   193 0.03206917 0.2050420 0.02458126 0.001743709 0.04006825 0.0010159651
## 194   194 0.03206973 0.2050280 0.02458116 0.001742525 0.04005330 0.0010148770
## 195   195 0.03207123 0.2049603 0.02458130 0.001741862 0.04000480 0.0010143535
## 196   196 0.03207069 0.2049848 0.02458195 0.001742802 0.04005671 0.0010153305
## 197   197 0.03206869 0.2050826 0.02457892 0.001745123 0.04013938 0.0010161935
## 198   198 0.03206907 0.2050597 0.02457929 0.001745878 0.04017312 0.0010187940
## 199   199 0.03206995 0.2050216 0.02458109 0.001746296 0.04016722 0.0010193117
## 200   200 0.03206896 0.2050647 0.02457954 0.001748017 0.04025962 0.0010197098
## 201   201 0.03206883 0.2050716 0.02457940 0.001748384 0.04024545 0.0010207262
## 202   202 0.03206772 0.2051314 0.02457711 0.001749645 0.04029812 0.0010228163
## 203   203 0.03206745 0.2051314 0.02457619 0.001744907 0.04014658 0.0010186347
## 204   204 0.03206770 0.2051181 0.02457536 0.001745263 0.04015378 0.0010182995
## 205   205 0.03206632 0.2051692 0.02457445 0.001742740 0.04006480 0.0010177920
## 206   206 0.03206751 0.2051218 0.02457500 0.001743598 0.04009224 0.0010198099
## 207   207 0.03206897 0.2050495 0.02457577 0.001741183 0.04002018 0.0010176988
## 208   208 0.03206968 0.2050207 0.02457615 0.001739870 0.03996805 0.0010155763
## 209   209 0.03206953 0.2050276 0.02457688 0.001738998 0.03991646 0.0010149879
## 210   210 0.03207019 0.2050072 0.02457752 0.001739041 0.03990977 0.0010156110
## 211   211 0.03207051 0.2049941 0.02457788 0.001736662 0.03982307 0.0010140456
## 212   212 0.03207207 0.2049278 0.02457917 0.001735638 0.03979933 0.0010141485
## 213   213 0.03207192 0.2049312 0.02457959 0.001734728 0.03977390 0.0010133716
## 214   214 0.03207326 0.2048654 0.02458110 0.001732864 0.03970858 0.0010122239
## 215   215 0.03207362 0.2048447 0.02458269 0.001731383 0.03964952 0.0010114171
## 216   216 0.03207405 0.2048244 0.02458258 0.001730747 0.03962130 0.0010105264
## 217   217 0.03207360 0.2048493 0.02458291 0.001730862 0.03962953 0.0010117337
## 218   218 0.03207367 0.2048519 0.02458347 0.001732159 0.03967375 0.0010137218
## 219   219 0.03207287 0.2048856 0.02458252 0.001731865 0.03966546 0.0010142560
## 220   220 0.03207280 0.2048914 0.02458221 0.001731415 0.03964697 0.0010147703
## 221   221 0.03207320 0.2048780 0.02458173 0.001731722 0.03966011 0.0010147500
## 222   222 0.03207325 0.2048653 0.02458192 0.001730936 0.03963943 0.0010142181
## 223   223 0.03207242 0.2049032 0.02458086 0.001732350 0.03968642 0.0010157095
## 224   224 0.03207233 0.2049031 0.02458054 0.001731883 0.03966506 0.0010149805
## 225   225 0.03207208 0.2049147 0.02458105 0.001731677 0.03965140 0.0010147254
## 226   226 0.03207187 0.2049254 0.02458114 0.001731322 0.03964074 0.0010143681
## 227   227 0.03207190 0.2049216 0.02458136 0.001730803 0.03962087 0.0010141621
## 228   228 0.03207191 0.2049236 0.02458172 0.001730910 0.03963172 0.0010141598
## 229   229 0.03207167 0.2049328 0.02458197 0.001730707 0.03962512 0.0010136789
## 230   230 0.03207144 0.2049413 0.02458189 0.001730667 0.03962483 0.0010132987
## 231   231 0.03207139 0.2049460 0.02458187 0.001730581 0.03962191 0.0010130583
## 232   232 0.03207141 0.2049454 0.02458210 0.001730423 0.03961255 0.0010130997
## 233   233 0.03207113 0.2049573 0.02458187 0.001730504 0.03961599 0.0010128952
## 234   234 0.03207138 0.2049481 0.02458204 0.001730713 0.03961909 0.0010130771
## 235   235 0.03207144 0.2049436 0.02458214 0.001730248 0.03960528 0.0010129099
## 236   236 0.03207155 0.2049378 0.02458225 0.001729964 0.03959589 0.0010128528
## 237   237 0.03207166 0.2049333 0.02458232 0.001730128 0.03959893 0.0010132226
## 238   238 0.03207158 0.2049367 0.02458219 0.001730269 0.03960374 0.0010132711
## 239   239 0.03207162 0.2049353 0.02458225 0.001730332 0.03960637 0.0010133297
## 240   240 0.03207168 0.2049328 0.02458231 0.001730399 0.03960843 0.0010134866
## [1] "Best Model"
##   nvmax
## 9     9

## `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.

## [1] "Coefficients of final model:"
##                  Estimate         2.5 %        97.5 %
## (Intercept)  1.997430e+00  1.990899e+00  2.003962e+00
## x4          -5.115588e-05 -6.853808e-05 -3.377368e-05
## x7           1.102188e-02  9.794023e-03  1.224974e-02
## x9           3.070804e-03  2.432449e-03  3.709159e-03
## x10          1.278875e-03  6.873502e-04  1.870401e-03
## x16          9.700205e-04  5.568889e-04  1.383152e-03
## x17          1.600779e-03  9.763881e-04  2.225170e-03
## stat98       3.343631e-03  2.875903e-03  3.811359e-03
## stat110     -3.137873e-03 -3.613099e-03 -2.662647e-03
## x18.sqrt     2.631786e-02  2.450088e-02  2.813484e-02

Test

if (algo.forward.caret == TRUE){
    test.model(model=model.forward, test=data.test
             ,method = 'leapForward',subopt = NULL
             ,formula = formula, feature.names = feature.names, label.names = label.names
             ,id = id
             ,draw.limits = TRUE, transformation = t)
}

## [1] "Summary of predicted values: "
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##   2.044   2.084   2.097   2.096   2.108   2.145 
## [1] "leapForward  Test MSE: 0.00104102201936567"

Backward Elimination with CV

Train

if (algo.backward.caret == TRUE){
  set.seed(1)
  returned = train.caret.glmselect(formula = formula
                                   ,data =  data.train
                                   ,method = "leapBackward"
                                   ,feature.names =  feature.names)
  model.backward = returned$model
  id = returned$id
}

## Aggregating results
## Selecting tuning parameters
## Fitting nvmax = 9 on full training set
## [1] "All models results"
##     nvmax       RMSE  Rsquared        MAE      RMSESD RsquaredSD        MAESD
## 1       1 0.03376443 0.1132051 0.02637850 0.001210125 0.02571127 0.0007953495
## 2       2 0.03297241 0.1543836 0.02566825 0.001331813 0.02849647 0.0008195492
## 3       3 0.03250196 0.1779147 0.02515184 0.001317023 0.02808594 0.0007621952
## 4       4 0.03196246 0.2052676 0.02446499 0.001452681 0.03363383 0.0008687720
## 5       5 0.03171086 0.2179498 0.02429750 0.001512147 0.03600988 0.0008518664
## 6       6 0.03165522 0.2206821 0.02425311 0.001516391 0.03578302 0.0008635422
## 7       7 0.03161930 0.2224386 0.02422076 0.001591134 0.03757869 0.0009392440
## 8       8 0.03155244 0.2257309 0.02419734 0.001617014 0.03868599 0.0009624076
## 9       9 0.03146134 0.2301155 0.02411499 0.001613656 0.03869431 0.0009595924
## 10     10 0.03148846 0.2287059 0.02413610 0.001591183 0.03762129 0.0009509402
## 11     11 0.03149049 0.2285003 0.02414479 0.001570434 0.03691604 0.0009354630
## 12     12 0.03150288 0.2280520 0.02416044 0.001590991 0.03717482 0.0009408043
## 13     13 0.03147622 0.2293254 0.02413800 0.001586666 0.03717877 0.0009728358
## 14     14 0.03148550 0.2289291 0.02413604 0.001600811 0.03767695 0.0009991203
## 15     15 0.03151049 0.2277667 0.02415137 0.001606362 0.03779119 0.0009983448
## 16     16 0.03151635 0.2273578 0.02416525 0.001572932 0.03632568 0.0009714183
## 17     17 0.03151854 0.2274304 0.02417083 0.001620733 0.03851447 0.0010107519
## 18     18 0.03151706 0.2274435 0.02417252 0.001612832 0.03798687 0.0010024230
## 19     19 0.03150643 0.2279334 0.02417277 0.001614284 0.03758714 0.0009928002
## 20     20 0.03151455 0.2275297 0.02416644 0.001601300 0.03679747 0.0009782668
## 21     21 0.03152350 0.2272073 0.02417310 0.001616084 0.03729289 0.0009942840
## 22     22 0.03153133 0.2268917 0.02418051 0.001637620 0.03791465 0.0010081285
## 23     23 0.03153472 0.2267931 0.02419805 0.001654486 0.03865559 0.0010016789
## 24     24 0.03154107 0.2265226 0.02418155 0.001654418 0.03873667 0.0009895126
## 25     25 0.03155055 0.2260937 0.02418320 0.001655011 0.03888933 0.0009942238
## 26     26 0.03156946 0.2253051 0.02418445 0.001671881 0.03942483 0.0010103395
## 27     27 0.03156320 0.2255405 0.02417591 0.001656519 0.03929361 0.0010097896
## 28     28 0.03155968 0.2256921 0.02416543 0.001641156 0.03889225 0.0009961736
## 29     29 0.03156008 0.2257004 0.02417187 0.001632050 0.03851225 0.0009880978
## 30     30 0.03155303 0.2260316 0.02415863 0.001638438 0.03866867 0.0010092221
## 31     31 0.03154030 0.2265958 0.02415053 0.001633024 0.03834113 0.0010009403
## 32     32 0.03151614 0.2277098 0.02413535 0.001635320 0.03874525 0.0010068959
## 33     33 0.03151612 0.2277270 0.02413915 0.001633799 0.03880826 0.0009846613
## 34     34 0.03152601 0.2273383 0.02414968 0.001637943 0.03907524 0.0009900342
## 35     35 0.03155330 0.2260819 0.02416139 0.001631636 0.03882983 0.0009646840
## 36     36 0.03156681 0.2254905 0.02417513 0.001634204 0.03899492 0.0009505969
## 37     37 0.03156984 0.2253640 0.02417301 0.001627327 0.03899171 0.0009414108
## 38     38 0.03158292 0.2247439 0.02417679 0.001626142 0.03852619 0.0009428337
## 39     39 0.03159319 0.2242198 0.02418826 0.001606407 0.03783241 0.0009290149
## 40     40 0.03159669 0.2240627 0.02418838 0.001605620 0.03793335 0.0009231437
## 41     41 0.03159141 0.2243412 0.02418673 0.001597770 0.03758194 0.0009218706
## 42     42 0.03160271 0.2238269 0.02420513 0.001602599 0.03744318 0.0009346345
## 43     43 0.03162291 0.2229319 0.02422565 0.001613625 0.03779303 0.0009251648
## 44     44 0.03161870 0.2232648 0.02422728 0.001634643 0.03844407 0.0009409630
## 45     45 0.03162575 0.2229534 0.02423617 0.001629968 0.03831795 0.0009594620
## 46     46 0.03164307 0.2222409 0.02424256 0.001646312 0.03900960 0.0009557584
## 47     47 0.03164253 0.2222972 0.02424679 0.001650969 0.03910748 0.0009549745
## 48     48 0.03164846 0.2220330 0.02424821 0.001642169 0.03838372 0.0009513925
## 49     49 0.03164170 0.2223116 0.02424750 0.001632779 0.03832009 0.0009456064
## 50     50 0.03164476 0.2222214 0.02425721 0.001642346 0.03859620 0.0009515632
## 51     51 0.03165193 0.2219214 0.02425602 0.001652095 0.03874678 0.0009640624
## 52     52 0.03166112 0.2215574 0.02425967 0.001665668 0.03901083 0.0009741897
## 53     53 0.03165650 0.2218185 0.02425549 0.001668125 0.03904165 0.0009865959
## 54     54 0.03166885 0.2213211 0.02426931 0.001679554 0.03956173 0.0009859223
## 55     55 0.03168163 0.2207730 0.02428046 0.001674059 0.03929660 0.0009798176
## 56     56 0.03168683 0.2205238 0.02428724 0.001666329 0.03898309 0.0009758769
## 57     57 0.03169172 0.2202992 0.02429299 0.001660758 0.03877914 0.0009719258
## 58     58 0.03170084 0.2199577 0.02430556 0.001680169 0.03936789 0.0009869420
## 59     59 0.03170868 0.2195765 0.02430840 0.001686251 0.03970138 0.0009934343
## 60     60 0.03171373 0.2193797 0.02430721 0.001685401 0.03967520 0.0009924490
## 61     61 0.03172095 0.2190824 0.02430536 0.001683846 0.03983444 0.0009859919
## 62     62 0.03173676 0.2183728 0.02431540 0.001676745 0.03924955 0.0009840450
## 63     63 0.03173566 0.2184635 0.02431800 0.001675287 0.03931261 0.0009796601
## 64     64 0.03174800 0.2179168 0.02433325 0.001663164 0.03862511 0.0009698114
## 65     65 0.03175247 0.2177127 0.02433372 0.001664481 0.03870935 0.0009669790
## 66     66 0.03177087 0.2168375 0.02434812 0.001652297 0.03786732 0.0009683765
## 67     67 0.03177685 0.2166231 0.02435198 0.001667331 0.03838883 0.0009768701
## 68     68 0.03177916 0.2165151 0.02434921 0.001653032 0.03788144 0.0009635538
## 69     69 0.03178149 0.2164660 0.02435498 0.001647806 0.03762553 0.0009509664
## 70     70 0.03178028 0.2165100 0.02436229 0.001640233 0.03734418 0.0009446900
## 71     71 0.03179564 0.2158159 0.02437843 0.001632269 0.03710029 0.0009381754
## 72     72 0.03180255 0.2155075 0.02437518 0.001632268 0.03687301 0.0009382289
## 73     73 0.03180341 0.2154291 0.02437042 0.001613656 0.03618860 0.0009237270
## 74     74 0.03180664 0.2153014 0.02438649 0.001615000 0.03627773 0.0009253223
## 75     75 0.03181452 0.2149576 0.02439193 0.001614592 0.03640882 0.0009285769
## 76     76 0.03183510 0.2140674 0.02441256 0.001625175 0.03662858 0.0009327369
## 77     77 0.03183378 0.2141269 0.02440372 0.001624565 0.03641434 0.0009283979
## 78     78 0.03184326 0.2137857 0.02440586 0.001636562 0.03695941 0.0009348021
## 79     79 0.03184414 0.2137899 0.02439668 0.001632421 0.03674510 0.0009286141
## 80     80 0.03184240 0.2138410 0.02439629 0.001623171 0.03645926 0.0009226085
## 81     81 0.03184355 0.2138035 0.02439771 0.001630765 0.03664124 0.0009296515
## 82     82 0.03185316 0.2133683 0.02440806 0.001627985 0.03638365 0.0009316153
## 83     83 0.03185331 0.2134055 0.02440493 0.001626828 0.03621102 0.0009318230
## 84     84 0.03185812 0.2131926 0.02441273 0.001621493 0.03605908 0.0009229377
## 85     85 0.03185973 0.2131309 0.02441101 0.001615861 0.03568937 0.0009215410
## 86     86 0.03187074 0.2126741 0.02442318 0.001622649 0.03604431 0.0009273429
## 87     87 0.03187398 0.2125347 0.02442644 0.001628812 0.03630578 0.0009341526
## 88     88 0.03187304 0.2125894 0.02442682 0.001627924 0.03615784 0.0009347815
## 89     89 0.03186499 0.2129482 0.02442956 0.001620726 0.03615265 0.0009321775
## 90     90 0.03187006 0.2127345 0.02443499 0.001619794 0.03599117 0.0009368625
## 91     91 0.03187912 0.2123591 0.02444726 0.001627911 0.03637699 0.0009475462
## 92     92 0.03188901 0.2119675 0.02445430 0.001633350 0.03661004 0.0009471446
## 93     93 0.03190221 0.2114126 0.02446904 0.001639925 0.03685801 0.0009531317
## 94     94 0.03191062 0.2110209 0.02447440 0.001637373 0.03671421 0.0009500144
## 95     95 0.03191444 0.2108484 0.02447644 0.001643076 0.03686508 0.0009497913
## 96     96 0.03193166 0.2101177 0.02449032 0.001649354 0.03697641 0.0009477463
## 97     97 0.03193476 0.2100126 0.02449239 0.001646448 0.03665186 0.0009469300
## 98     98 0.03193593 0.2099594 0.02449311 0.001642868 0.03637415 0.0009446989
## 99     99 0.03194040 0.2098085 0.02450478 0.001648869 0.03662896 0.0009504618
## 100   100 0.03194567 0.2095663 0.02451329 0.001653175 0.03667706 0.0009473889
## 101   101 0.03194332 0.2096481 0.02450551 0.001654923 0.03669238 0.0009494527
## 102   102 0.03195094 0.2093602 0.02450876 0.001658724 0.03680229 0.0009459566
## 103   103 0.03195150 0.2093222 0.02451137 0.001656284 0.03683249 0.0009426637
## 104   104 0.03194431 0.2096316 0.02450884 0.001659702 0.03693890 0.0009467433
## 105   105 0.03193967 0.2098606 0.02450051 0.001665449 0.03727246 0.0009495248
## 106   106 0.03194176 0.2098140 0.02450074 0.001675530 0.03763435 0.0009585169
## 107   107 0.03194515 0.2097282 0.02450124 0.001681835 0.03782683 0.0009629241
## 108   108 0.03195052 0.2095448 0.02450451 0.001690632 0.03801944 0.0009753126
## 109   109 0.03195337 0.2094251 0.02450302 0.001685034 0.03780720 0.0009707768
## 110   110 0.03196040 0.2090924 0.02450908 0.001677944 0.03751180 0.0009695303
## 111   111 0.03196052 0.2091093 0.02450686 0.001679228 0.03740028 0.0009686649
## 112   112 0.03196368 0.2090227 0.02451285 0.001685651 0.03759708 0.0009712771
## 113   113 0.03195830 0.2092894 0.02451325 0.001683738 0.03769225 0.0009717533
## 114   114 0.03196448 0.2090170 0.02451843 0.001682308 0.03764379 0.0009729096
## 115   115 0.03196050 0.2092388 0.02451512 0.001686512 0.03767290 0.0009736178
## 116   116 0.03196352 0.2091128 0.02451980 0.001686154 0.03767998 0.0009740847
## 117   117 0.03196711 0.2089605 0.02452610 0.001684090 0.03757049 0.0009708766
## 118   118 0.03196965 0.2088583 0.02452959 0.001679011 0.03733576 0.0009670367
## 119   119 0.03196947 0.2088370 0.02452608 0.001673010 0.03707106 0.0009615711
## 120   120 0.03197392 0.2086510 0.02452973 0.001668032 0.03684659 0.0009593368
## 121   121 0.03197411 0.2086483 0.02452961 0.001669335 0.03690322 0.0009593472
## 122   122 0.03197685 0.2085641 0.02453581 0.001674711 0.03712505 0.0009689956
## 123   123 0.03198324 0.2083040 0.02453964 0.001678748 0.03737788 0.0009746737
## 124   124 0.03199337 0.2078714 0.02454828 0.001679221 0.03730500 0.0009737390
## 125   125 0.03199364 0.2078678 0.02454760 0.001682188 0.03744509 0.0009750597
## 126   126 0.03199575 0.2077564 0.02455053 0.001682908 0.03740870 0.0009767111
## 127   127 0.03199795 0.2076857 0.02454423 0.001681225 0.03736124 0.0009751879
## 128   128 0.03200211 0.2075362 0.02454392 0.001690064 0.03767706 0.0009786170
## 129   129 0.03200455 0.2074668 0.02454282 0.001694358 0.03793396 0.0009826791
## 130   130 0.03200390 0.2075405 0.02454280 0.001698883 0.03815146 0.0009839730
## 131   131 0.03200625 0.2074741 0.02454332 0.001708004 0.03846112 0.0009900765
## 132   132 0.03200289 0.2076352 0.02454190 0.001716637 0.03870799 0.0009984791
## 133   133 0.03200790 0.2074340 0.02454655 0.001715338 0.03865955 0.0009919718
## 134   134 0.03201373 0.2072098 0.02455047 0.001718614 0.03884604 0.0009939049
## 135   135 0.03201610 0.2071012 0.02455662 0.001718211 0.03883545 0.0009972249
## 136   136 0.03202107 0.2069231 0.02456087 0.001723037 0.03895849 0.0010002304
## 137   137 0.03202205 0.2068609 0.02456044 0.001717656 0.03881638 0.0009982568
## 138   138 0.03202430 0.2067610 0.02456189 0.001715968 0.03875214 0.0009935655
## 139   139 0.03203175 0.2064532 0.02456804 0.001720138 0.03889162 0.0009923886
## 140   140 0.03203003 0.2065156 0.02456911 0.001720819 0.03906064 0.0009986241
## 141   141 0.03203095 0.2064841 0.02456690 0.001721875 0.03917149 0.0009975847
## 142   142 0.03203701 0.2062293 0.02457037 0.001721090 0.03918567 0.0010013638
## 143   143 0.03203248 0.2064534 0.02456847 0.001727144 0.03939548 0.0010094938
## 144   144 0.03203004 0.2065512 0.02456533 0.001725724 0.03935372 0.0010066377
## 145   145 0.03202781 0.2066199 0.02456512 0.001723152 0.03920696 0.0010072855
## 146   146 0.03202867 0.2065976 0.02456735 0.001727172 0.03933727 0.0010101899
## 147   147 0.03203087 0.2065116 0.02456887 0.001728624 0.03941244 0.0010122912
## 148   148 0.03203364 0.2064273 0.02457026 0.001732490 0.03958166 0.0010147771
## 149   149 0.03204024 0.2061508 0.02457446 0.001731066 0.03947658 0.0010145296
## 150   150 0.03204280 0.2060230 0.02457340 0.001730403 0.03942516 0.0010151918
## 151   151 0.03204141 0.2060872 0.02457115 0.001734455 0.03957117 0.0010218562
## 152   152 0.03204300 0.2060226 0.02456864 0.001733689 0.03946074 0.0010174451
## 153   153 0.03204505 0.2059347 0.02457140 0.001732184 0.03939813 0.0010130176
## 154   154 0.03204641 0.2058849 0.02457278 0.001729495 0.03940792 0.0010148582
## 155   155 0.03204793 0.2058252 0.02457399 0.001728896 0.03940343 0.0010122740
## 156   156 0.03205108 0.2056920 0.02457397 0.001729682 0.03940055 0.0010073055
## 157   157 0.03204971 0.2057734 0.02457339 0.001734533 0.03956170 0.0010095181
## 158   158 0.03204901 0.2058135 0.02457365 0.001736523 0.03964971 0.0010096661
## 159   159 0.03204906 0.2058248 0.02457177 0.001737911 0.03966948 0.0010102455
## 160   160 0.03204874 0.2058629 0.02457097 0.001737810 0.03965245 0.0010125979
## 161   161 0.03205296 0.2056876 0.02457500 0.001738080 0.03962469 0.0010101045
## 162   162 0.03205761 0.2055122 0.02457869 0.001742117 0.03975448 0.0010109296
## 163   163 0.03206178 0.2053210 0.02458069 0.001742289 0.03971724 0.0010066157
## 164   164 0.03205688 0.2055257 0.02457507 0.001742000 0.03979812 0.0010101092
## 165   165 0.03205838 0.2054506 0.02457697 0.001740489 0.03972717 0.0010073238
## 166   166 0.03205829 0.2054295 0.02457574 0.001738206 0.03969447 0.0010085867
## 167   167 0.03205831 0.2054351 0.02457826 0.001735121 0.03961186 0.0010087757
## 168   168 0.03205708 0.2055010 0.02457956 0.001737201 0.03969842 0.0010108297
## 169   169 0.03205765 0.2054833 0.02458242 0.001737848 0.03975777 0.0010095326
## 170   170 0.03205692 0.2055109 0.02457902 0.001739632 0.03986380 0.0010096672
## 171   171 0.03205840 0.2054563 0.02457740 0.001741787 0.03998514 0.0010116398
## 172   172 0.03206066 0.2053646 0.02457968 0.001746255 0.04023872 0.0010136217
## 173   173 0.03206337 0.2052442 0.02458318 0.001747884 0.04023816 0.0010178862
## 174   174 0.03206302 0.2052716 0.02458261 0.001750363 0.04035796 0.0010179336
## 175   175 0.03206267 0.2052794 0.02458317 0.001748168 0.04029320 0.0010135216
## 176   176 0.03206449 0.2052126 0.02458324 0.001749031 0.04023667 0.0010146625
## 177   177 0.03206468 0.2051944 0.02458241 0.001745893 0.04014386 0.0010125679
## 178   178 0.03206690 0.2050950 0.02458418 0.001746059 0.04020591 0.0010140861
## 179   179 0.03206963 0.2049935 0.02458641 0.001744099 0.04012297 0.0010125982
## 180   180 0.03206868 0.2050381 0.02458600 0.001747179 0.04027861 0.0010136997
## 181   181 0.03206663 0.2051288 0.02458658 0.001745268 0.04023468 0.0010138553
## 182   182 0.03206500 0.2051892 0.02458547 0.001743100 0.04014883 0.0010145327
## 183   183 0.03206835 0.2050578 0.02458614 0.001743329 0.04016153 0.0010144200
## 184   184 0.03206652 0.2051432 0.02458384 0.001742172 0.04007395 0.0010134162
## 185   185 0.03206504 0.2052120 0.02458092 0.001742836 0.04005249 0.0010144394
## 186   186 0.03206671 0.2051392 0.02458023 0.001741803 0.04000189 0.0010129632
## 187   187 0.03206473 0.2052182 0.02457762 0.001738734 0.03994668 0.0010113227
## 188   188 0.03206629 0.2051647 0.02457988 0.001741694 0.04002117 0.0010136551
## 189   189 0.03206730 0.2051053 0.02458020 0.001741748 0.04005787 0.0010148205
## 190   190 0.03206552 0.2051807 0.02457924 0.001743614 0.04011565 0.0010143605
## 191   191 0.03206896 0.2050331 0.02458283 0.001744766 0.04012691 0.0010136410
## 192   192 0.03207025 0.2049907 0.02458290 0.001744863 0.04009111 0.0010145145
## 193   193 0.03206830 0.2050794 0.02458018 0.001743904 0.04007501 0.0010155272
## 194   194 0.03206718 0.2051318 0.02457819 0.001743454 0.04007019 0.0010162075
## 195   195 0.03206803 0.2050948 0.02457807 0.001743314 0.04006339 0.0010159525
## 196   196 0.03206899 0.2050562 0.02457941 0.001743544 0.04008443 0.0010166987
## 197   197 0.03206869 0.2050826 0.02457892 0.001745123 0.04013938 0.0010161935
## 198   198 0.03206855 0.2050853 0.02457905 0.001745915 0.04016540 0.0010189015
## 199   199 0.03206919 0.2050596 0.02458065 0.001746714 0.04018099 0.0010193463
## 200   200 0.03206777 0.2051261 0.02457848 0.001748371 0.04026181 0.0010201048
## 201   201 0.03206832 0.2050969 0.02457912 0.001748421 0.04023792 0.0010208528
## 202   202 0.03206772 0.2051314 0.02457711 0.001749645 0.04029812 0.0010228163
## 203   203 0.03206745 0.2051314 0.02457619 0.001744907 0.04014658 0.0010186347
## 204   204 0.03206800 0.2051065 0.02457627 0.001745241 0.04015729 0.0010178985
## 205   205 0.03206728 0.2051326 0.02457580 0.001742674 0.04007599 0.0010171964
## 206   206 0.03206777 0.2051166 0.02457590 0.001743580 0.04009384 0.0010194146
## 207   207 0.03206837 0.2050758 0.02457590 0.001741224 0.04001211 0.0010176408
## 208   208 0.03206968 0.2050207 0.02457615 0.001739870 0.03996805 0.0010155763
## 209   209 0.03206953 0.2050276 0.02457688 0.001738998 0.03991646 0.0010149879
## 210   210 0.03207019 0.2050072 0.02457752 0.001739041 0.03990977 0.0010156110
## 211   211 0.03207051 0.2049941 0.02457788 0.001736662 0.03982307 0.0010140456
## 212   212 0.03207207 0.2049278 0.02457917 0.001735638 0.03979933 0.0010141485
## 213   213 0.03207192 0.2049312 0.02457959 0.001734728 0.03977390 0.0010133716
## 214   214 0.03207326 0.2048654 0.02458110 0.001732864 0.03970858 0.0010122239
## 215   215 0.03207362 0.2048447 0.02458269 0.001731383 0.03964952 0.0010114171
## 216   216 0.03207405 0.2048244 0.02458258 0.001730747 0.03962130 0.0010105264
## 217   217 0.03207360 0.2048493 0.02458291 0.001730862 0.03962953 0.0010117337
## 218   218 0.03207367 0.2048519 0.02458347 0.001732159 0.03967375 0.0010137218
## 219   219 0.03207287 0.2048856 0.02458252 0.001731865 0.03966546 0.0010142560
## 220   220 0.03207260 0.2048978 0.02458218 0.001731602 0.03965058 0.0010148007
## 221   221 0.03207300 0.2048841 0.02458170 0.001731904 0.03966357 0.0010147819
## 222   222 0.03207325 0.2048653 0.02458192 0.001730936 0.03963943 0.0010142181
## 223   223 0.03207242 0.2049032 0.02458086 0.001732350 0.03968642 0.0010157095
## 224   224 0.03207233 0.2049031 0.02458054 0.001731883 0.03966506 0.0010149805
## 225   225 0.03207208 0.2049147 0.02458105 0.001731677 0.03965140 0.0010147254
## 226   226 0.03207187 0.2049254 0.02458114 0.001731322 0.03964074 0.0010143681
## 227   227 0.03207190 0.2049216 0.02458136 0.001730803 0.03962087 0.0010141621
## 228   228 0.03207191 0.2049236 0.02458172 0.001730910 0.03963172 0.0010141598
## 229   229 0.03207167 0.2049328 0.02458197 0.001730707 0.03962512 0.0010136789
## 230   230 0.03207144 0.2049413 0.02458189 0.001730667 0.03962483 0.0010132987
## 231   231 0.03207139 0.2049460 0.02458187 0.001730581 0.03962191 0.0010130583
## 232   232 0.03207141 0.2049454 0.02458210 0.001730423 0.03961255 0.0010130997
## 233   233 0.03207113 0.2049573 0.02458187 0.001730504 0.03961599 0.0010128952
## 234   234 0.03207138 0.2049481 0.02458204 0.001730713 0.03961909 0.0010130771
## 235   235 0.03207144 0.2049436 0.02458214 0.001730248 0.03960528 0.0010129099
## 236   236 0.03207155 0.2049378 0.02458225 0.001729964 0.03959589 0.0010128528
## 237   237 0.03207166 0.2049333 0.02458232 0.001730128 0.03959893 0.0010132226
## 238   238 0.03207158 0.2049367 0.02458219 0.001730269 0.03960374 0.0010132711
## 239   239 0.03207162 0.2049353 0.02458225 0.001730332 0.03960637 0.0010133297
## 240   240 0.03207168 0.2049328 0.02458231 0.001730399 0.03960843 0.0010134866
## [1] "Best Model"
##   nvmax
## 9     9

## `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.

## [1] "Coefficients of final model:"
##                  Estimate         2.5 %        97.5 %
## (Intercept)  1.997430e+00  1.990899e+00  2.003962e+00
## x4          -5.115588e-05 -6.853808e-05 -3.377368e-05
## x7           1.102188e-02  9.794023e-03  1.224974e-02
## x9           3.070804e-03  2.432449e-03  3.709159e-03
## x10          1.278875e-03  6.873502e-04  1.870401e-03
## x16          9.700205e-04  5.568889e-04  1.383152e-03
## x17          1.600779e-03  9.763881e-04  2.225170e-03
## stat98       3.343631e-03  2.875903e-03  3.811359e-03
## stat110     -3.137873e-03 -3.613099e-03 -2.662647e-03
## x18.sqrt     2.631786e-02  2.450088e-02  2.813484e-02

Test

if (algo.backward.caret == TRUE){
  test.model(model.backward, data.test
             ,method = 'leapBackward',subopt = NULL
             ,formula = formula, feature.names = feature.names, label.names = label.names
             ,id = id
             ,draw.limits = TRUE, transformation = t)
}

## [1] "Summary of predicted values: "
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##   2.044   2.084   2.097   2.096   2.108   2.145 
## [1] "leapBackward  Test MSE: 0.00104102201936567"

Stepwise Selection with CV

Train

if (algo.stepwise.caret == TRUE){
  set.seed(1)
  returned = train.caret.glmselect(formula = formula
                                   ,data =  data.train
                                   ,method = "leapSeq"
                                   ,feature.names = feature.names)
  model.stepwise = returned$model
  id = returned$id
}

## Aggregating results
## Selecting tuning parameters
## Fitting nvmax = 9 on full training set
## [1] "All models results"
##     nvmax       RMSE  Rsquared        MAE      RMSESD RsquaredSD        MAESD
## 1       1 0.03376443 0.1132051 0.02637850 0.001210125 0.02571127 0.0007953495
## 2       2 0.03297241 0.1543836 0.02566825 0.001331813 0.02849647 0.0008195492
## 3       3 0.03250196 0.1779147 0.02515184 0.001317023 0.02808594 0.0007621952
## 4       4 0.03196246 0.2052676 0.02446499 0.001452681 0.03363383 0.0008687720
## 5       5 0.03171086 0.2179498 0.02429750 0.001512147 0.03600988 0.0008518664
## 6       6 0.03165522 0.2206821 0.02425311 0.001516391 0.03578302 0.0008635422
## 7       7 0.03161930 0.2224386 0.02422076 0.001591134 0.03757869 0.0009392440
## 8       8 0.03155244 0.2257309 0.02419734 0.001617014 0.03868599 0.0009624076
## 9       9 0.03146134 0.2301155 0.02411499 0.001613656 0.03869431 0.0009595924
## 10     10 0.03148846 0.2287059 0.02413610 0.001591183 0.03762129 0.0009509402
## 11     11 0.03149049 0.2285003 0.02414479 0.001570434 0.03691604 0.0009354630
## 12     12 0.03150288 0.2280520 0.02416044 0.001590991 0.03717482 0.0009408043
## 13     13 0.03147622 0.2293254 0.02413800 0.001586666 0.03717877 0.0009728358
## 14     14 0.03148550 0.2289291 0.02413604 0.001600811 0.03767695 0.0009991203
## 15     15 0.03151049 0.2277667 0.02415137 0.001606362 0.03779119 0.0009983448
## 16     16 0.03151635 0.2273578 0.02416525 0.001572932 0.03632568 0.0009714183
## 17     17 0.03150438 0.2281246 0.02415982 0.001625619 0.03860212 0.0010191859
## 18     18 0.03150374 0.2280915 0.02415855 0.001606877 0.03802253 0.0009915896
## 19     19 0.03180657 0.2127004 0.02444456 0.001751595 0.05728238 0.0011105427
## 20     20 0.03152427 0.2270617 0.02418562 0.001595441 0.03650130 0.0009624108
## 21     21 0.03185903 0.2103130 0.02448502 0.002082927 0.06575323 0.0014950551
## 22     22 0.03182330 0.2118483 0.02444687 0.001776166 0.05860878 0.0011120743
## 23     23 0.03191553 0.2073860 0.02451267 0.001860581 0.06512860 0.0012336903
## 24     24 0.03216231 0.1944494 0.02463932 0.001641329 0.06674080 0.0011149214
## 25     25 0.03191805 0.2073050 0.02448142 0.001866018 0.06541768 0.0012364730
## 26     26 0.03178503 0.2144358 0.02440437 0.002304475 0.07058771 0.0016239364
## 27     27 0.03155685 0.2258517 0.02417755 0.001662713 0.03949810 0.0010084669
## 28     28 0.03250283 0.1774343 0.02491316 0.001775091 0.06989243 0.0012103644
## 29     29 0.03157131 0.2251786 0.02419074 0.001647132 0.03880779 0.0009873490
## 30     30 0.03154970 0.2261906 0.02415013 0.001629981 0.03831819 0.0009900989
## 31     31 0.03208839 0.1989632 0.02462543 0.002303147 0.07922805 0.0016665527
## 32     32 0.03151614 0.2277098 0.02413535 0.001635320 0.03874525 0.0010068959
## 33     33 0.03186205 0.2104133 0.02448265 0.002109339 0.06676413 0.0015193746
## 34     34 0.03152272 0.2274911 0.02414759 0.001641299 0.03917277 0.0009922418
## 35     35 0.03179208 0.2143355 0.02439784 0.002304906 0.07076244 0.0015970393
## 36     36 0.03157140 0.2252646 0.02418163 0.001636398 0.03897661 0.0009514999
## 37     37 0.03157625 0.2250578 0.02417752 0.001630306 0.03895796 0.0009419378
## 38     38 0.03186736 0.2096566 0.02438792 0.001595737 0.05633185 0.0009614105
## 39     39 0.03158770 0.2245021 0.02418861 0.001609478 0.03814285 0.0009410141
## 40     40 0.03158759 0.2245066 0.02417769 0.001612824 0.03837816 0.0009316826
## 41     41 0.03258460 0.1748103 0.02508744 0.002650055 0.09353770 0.0019488649
## 42     42 0.03219798 0.1928427 0.02469541 0.001840846 0.07329750 0.0013418269
## 43     43 0.03199373 0.2040535 0.02451406 0.001811164 0.06361541 0.0012098364
## 44     44 0.03162754 0.2228378 0.02423172 0.001627799 0.03786267 0.0009368672
## 45     45 0.03261552 0.1727649 0.02509820 0.002028939 0.07298912 0.0014640904
## 46     46 0.03164307 0.2222409 0.02424256 0.001646312 0.03900960 0.0009557584
## 47     47 0.03164358 0.2222103 0.02425024 0.001647030 0.03900757 0.0009616797
## 48     48 0.03198305 0.2051883 0.02451700 0.001709891 0.04970771 0.0010665078
## 49     49 0.03165174 0.2218470 0.02425662 0.001626083 0.03799849 0.0009448402
## 50     50 0.03226029 0.1915580 0.02477124 0.002389323 0.08322468 0.0017547306
## 51     51 0.03198749 0.2050767 0.02452486 0.001723179 0.04990481 0.0010724167
## 52     52 0.03199957 0.2039685 0.02452082 0.001794609 0.06288832 0.0013020712
## 53     53 0.03190377 0.2088868 0.02441388 0.002075109 0.06776368 0.0013079782
## 54     54 0.03166918 0.2212550 0.02427401 0.001658404 0.03866582 0.0009790142
## 55     55 0.03168320 0.2206958 0.02428269 0.001672702 0.03926647 0.0009780532
## 56     56 0.03192533 0.2091444 0.02451080 0.002332062 0.07020993 0.0015932784
## 57     57 0.03223547 0.1932815 0.02475608 0.002317976 0.07748192 0.0016013810
## 58     58 0.03240758 0.1841990 0.02485722 0.001820698 0.06445783 0.0012077502
## 59     59 0.03170542 0.2197424 0.02430603 0.001686600 0.03965226 0.0009944636
## 60     60 0.03205408 0.2024083 0.02464228 0.002125017 0.06580672 0.0014781327
## 61     61 0.03172205 0.2190406 0.02431334 0.001681907 0.03959259 0.0009946280
## 62     62 0.03228275 0.1892624 0.02475234 0.001756194 0.06971364 0.0011156835
## 63     63 0.03229511 0.1892865 0.02478744 0.001827147 0.06339599 0.0011415622
## 64     64 0.03196507 0.2062385 0.02446291 0.002009781 0.06403751 0.0012457144
## 65     65 0.03174477 0.2180366 0.02432846 0.001665271 0.03861272 0.0009678730
## 66     66 0.03263041 0.1733851 0.02505179 0.002308430 0.08286378 0.0016260834
## 67     67 0.03178364 0.2163051 0.02436123 0.001661866 0.03809429 0.0009711526
## 68     68 0.03206828 0.2011030 0.02457051 0.001612803 0.05403266 0.0009557886
## 69     69 0.03205619 0.2020661 0.02458725 0.001825042 0.06032267 0.0011363336
## 70     70 0.03178828 0.2161594 0.02435559 0.001639066 0.03728892 0.0009578125
## 71     71 0.03312894 0.1456091 0.02541524 0.002001845 0.08553175 0.0013312429
## 72     72 0.03179932 0.2156327 0.02437590 0.001630931 0.03683612 0.0009334147
## 73     73 0.03179394 0.2158327 0.02436494 0.001614773 0.03606989 0.0009250698
## 74     74 0.03238655 0.1868727 0.02494840 0.002582359 0.08224267 0.0018342461
## 75     75 0.03181196 0.2150635 0.02438655 0.001614466 0.03639353 0.0009293733
## 76     76 0.03248947 0.1811896 0.02498872 0.002053040 0.06534977 0.0013947616
## 77     77 0.03208432 0.2008153 0.02460894 0.001787448 0.05861012 0.0010997670
## 78     78 0.03208034 0.2008222 0.02461636 0.001778544 0.05801537 0.0010731560
## 79     79 0.03209338 0.2005167 0.02460375 0.001791165 0.05858244 0.0010995603
## 80     80 0.03184774 0.2136155 0.02440057 0.001627854 0.03655651 0.0009292682
## 81     81 0.03184270 0.2138395 0.02439748 0.001631264 0.03664300 0.0009271608
## 82     82 0.03209381 0.2019388 0.02463566 0.002303413 0.06792415 0.0015769932
## 83     83 0.03216917 0.1974000 0.02464777 0.001698973 0.04808826 0.0010144116
## 84     84 0.03245766 0.1824340 0.02486979 0.001721112 0.06358055 0.0011418707
## 85     85 0.03186362 0.2129756 0.02440530 0.001620473 0.03582436 0.0009230939
## 86     86 0.03211202 0.2012632 0.02464720 0.002304124 0.06770321 0.0015774051
## 87     87 0.03187583 0.2124513 0.02442006 0.001627270 0.03602309 0.0009293510
## 88     88 0.03246549 0.1820027 0.02498396 0.002129585 0.07292023 0.0014418823
## 89     89 0.03247020 0.1818635 0.02485957 0.002059604 0.07636237 0.0014919372
## 90     90 0.03223502 0.1943281 0.02471180 0.001785260 0.05945780 0.0011451141
## 91     91 0.03255707 0.1784624 0.02502766 0.002076190 0.06626388 0.0014327729
## 92     92 0.03212982 0.1992593 0.02460689 0.002005100 0.06356797 0.0012457591
## 93     93 0.03217219 0.1967780 0.02466662 0.001586315 0.05217334 0.0009411910
## 94     94 0.03216745 0.1977210 0.02466420 0.001618552 0.04793510 0.0010047667
## 95     95 0.03300307 0.1549856 0.02532738 0.002450403 0.09107113 0.0017126603
## 96     96 0.03248309 0.1809914 0.02494945 0.001792042 0.06209878 0.0010819688
## 97     97 0.03267572 0.1706223 0.02502078 0.001855528 0.07286906 0.0011916604
## 98     98 0.03242779 0.1840332 0.02487291 0.001739959 0.06302466 0.0011061861
## 99     99 0.03194607 0.2094921 0.02450579 0.001639222 0.03611328 0.0009399770
## 100   100 0.03194818 0.2093912 0.02451214 0.001644864 0.03632447 0.0009365280
## 101   101 0.03278677 0.1658663 0.02517803 0.002252411 0.08237153 0.0015313185
## 102   102 0.03195446 0.2092055 0.02451481 0.001657493 0.03674994 0.0009411005
## 103   103 0.03277694 0.1677600 0.02517840 0.002546771 0.09011683 0.0018986587
## 104   104 0.03226725 0.1931580 0.02475257 0.001720810 0.04884957 0.0010216540
## 105   105 0.03278503 0.1674423 0.02519923 0.002279650 0.08097792 0.0017560378
## 106   106 0.03218458 0.1964331 0.02472214 0.001799898 0.05793981 0.0010723418
## 107   107 0.03256156 0.1783155 0.02496798 0.001791113 0.06563365 0.0011764135
## 108   108 0.03347622 0.1311616 0.02572012 0.002059386 0.07824063 0.0015459613
## 109   109 0.03195263 0.2094592 0.02450485 0.001688301 0.03802164 0.0009730070
## 110   110 0.03219052 0.1981661 0.02474483 0.002318487 0.06760490 0.0016053914
## 111   111 0.03252334 0.1797951 0.02502890 0.002171106 0.07414936 0.0014721942
## 112   112 0.03221920 0.1956201 0.02469888 0.001673665 0.05038465 0.0010362436
## 113   113 0.03245085 0.1826447 0.02492555 0.001927148 0.07159623 0.0011925284
## 114   114 0.03219614 0.1961095 0.02472756 0.001815862 0.05818269 0.0010889246
## 115   115 0.03219454 0.1962164 0.02472537 0.001821613 0.05832753 0.0010923521
## 116   116 0.03254807 0.1787824 0.02496456 0.001855737 0.06436864 0.0011591968
## 117   117 0.03265948 0.1744696 0.02510710 0.002214367 0.07666527 0.0015401452
## 118   118 0.03246544 0.1840424 0.02495805 0.002298860 0.07376498 0.0016149487
## 119   119 0.03197181 0.2087687 0.02452912 0.001682093 0.03736782 0.0009753778
## 120   120 0.03197419 0.2086511 0.02453280 0.001672656 0.03703062 0.0009660818
## 121   121 0.03249295 0.1816352 0.02495147 0.001793204 0.06369034 0.0012480058
## 122   122 0.03243901 0.1846224 0.02488597 0.001984879 0.06641123 0.0012887717
## 123   123 0.03257969 0.1782496 0.02499821 0.001781955 0.06345946 0.0013035261
## 124   124 0.03215771 0.1995279 0.02468615 0.001624937 0.04081956 0.0009808812
## 125   125 0.03199166 0.2079617 0.02454281 0.001681791 0.03746445 0.0009744060
## 126   126 0.03259946 0.1774491 0.02501612 0.001789480 0.06352400 0.0012988877
## 127   127 0.03257950 0.1786850 0.02506272 0.002008896 0.06557183 0.0015250913
## 128   128 0.03199860 0.2076612 0.02454475 0.001680452 0.03738761 0.0009771339
## 129   129 0.03200553 0.2074243 0.02454265 0.001693590 0.03787085 0.0009830174
## 130   130 0.03237673 0.1892822 0.02490867 0.002197071 0.06478141 0.0015544970
## 131   131 0.03277095 0.1671905 0.02517120 0.001685743 0.06745602 0.0011780545
## 132   132 0.03200011 0.2077163 0.02453862 0.001702624 0.03822628 0.0009891626
## 133   133 0.03219649 0.1979206 0.02469256 0.001654623 0.03657975 0.0009397937
## 134   134 0.03200920 0.2073912 0.02454764 0.001716740 0.03869600 0.0009939137
## 135   135 0.03219794 0.1989377 0.02473962 0.002230471 0.06132236 0.0015091257
## 136   136 0.03232750 0.1904389 0.02479058 0.001930337 0.05960302 0.0012036057
## 137   137 0.03227318 0.1942744 0.02476719 0.001779307 0.05105972 0.0010981336
## 138   138 0.03284642 0.1648920 0.02516725 0.001620340 0.05606680 0.0011486641
## 139   139 0.03217182 0.1983762 0.02470763 0.001764266 0.04864377 0.0010315767
## 140   140 0.03222010 0.1969888 0.02471311 0.001673310 0.03731038 0.0009548168
## 141   141 0.03203115 0.2064620 0.02456672 0.001721914 0.03915652 0.0009991486
## 142   142 0.03254022 0.1812317 0.02499340 0.002274078 0.06577468 0.0015342045
## 143   143 0.03216936 0.1994328 0.02465016 0.001666891 0.04028133 0.0009878448
## 144   144 0.03203303 0.2064550 0.02456672 0.001733601 0.03956503 0.0010130549
## 145   145 0.03216576 0.1995329 0.02464865 0.001659829 0.04008492 0.0009840849
## 146   146 0.03234594 0.1906656 0.02488200 0.002254655 0.06646640 0.0015102680
## 147   147 0.03203257 0.2064462 0.02456749 0.001726234 0.03930241 0.0010109550
## 148   148 0.03277555 0.1701133 0.02519636 0.002235118 0.06151733 0.0015331982
## 149   149 0.03203933 0.2061971 0.02457127 0.001729333 0.03941595 0.0010133406
## 150   150 0.03252878 0.1818670 0.02500250 0.001988737 0.05969030 0.0013050212
## 151   151 0.03231986 0.1912536 0.02475953 0.001839744 0.05277372 0.0011442123
## 152   152 0.03270099 0.1726110 0.02508621 0.001586663 0.04943492 0.0009465519
## 153   153 0.03279616 0.1682314 0.02521524 0.002403424 0.07547047 0.0016466003
## 154   154 0.03204793 0.2058146 0.02457397 0.001728257 0.03930793 0.0010136622
## 155   155 0.03252093 0.1803958 0.02496550 0.001864033 0.05553328 0.0011233377
## 156   156 0.03204863 0.2057925 0.02457229 0.001730239 0.03938558 0.0010079784
## 157   157 0.03205065 0.2057295 0.02457438 0.001734087 0.03954065 0.0010090151
## 158   158 0.03224639 0.1956403 0.02473041 0.001803532 0.05167764 0.0010886356
## 159   159 0.03227897 0.1943842 0.02478841 0.001967456 0.05259696 0.0012494877
## 160   160 0.03243858 0.1857693 0.02488078 0.001949541 0.06107178 0.0012413248
## 161   161 0.03205295 0.2056610 0.02457585 0.001733653 0.03950760 0.0010093216
## 162   162 0.03205590 0.2055592 0.02457841 0.001738022 0.03963933 0.0010107676
## 163   163 0.03206084 0.2053693 0.02458082 0.001742692 0.03974019 0.0010066030
## 164   164 0.03225325 0.1971671 0.02476619 0.002283092 0.06228799 0.0015367419
## 165   165 0.03231614 0.1926420 0.02479039 0.001809921 0.05235767 0.0011203866
## 166   166 0.03246592 0.1844695 0.02488341 0.001585056 0.04394090 0.0008941183
## 167   167 0.03205903 0.2053964 0.02458083 0.001735062 0.03962292 0.0010076302
## 168   168 0.03251831 0.1823926 0.02495957 0.001849592 0.05998440 0.0011759527
## 169   169 0.03239734 0.1881578 0.02482491 0.001608404 0.03774759 0.0009333947
## 170   170 0.03222165 0.1963218 0.02474366 0.001802611 0.05136022 0.0010681783
## 171   171 0.03206131 0.2053458 0.02457973 0.001742161 0.03998391 0.0010115774
## 172   172 0.03228037 0.1938016 0.02474501 0.001678570 0.04933575 0.0009755835
## 173   173 0.03225715 0.1953247 0.02474400 0.001809714 0.05180095 0.0011032219
## 174   174 0.03249471 0.1828016 0.02495828 0.001791054 0.05808263 0.0012106012
## 175   175 0.03229939 0.1936205 0.02479921 0.001986360 0.05339896 0.0012528897
## 176   176 0.03240468 0.1892697 0.02487375 0.002433367 0.07169875 0.0016487235
## 177   177 0.03206516 0.2051777 0.02458373 0.001746388 0.04016510 0.0010142570
## 178   178 0.03206690 0.2050950 0.02458418 0.001746059 0.04020591 0.0010140861
## 179   179 0.03220469 0.1976533 0.02467090 0.001924977 0.05386313 0.0011476952
## 180   180 0.03206854 0.2050462 0.02458585 0.001747283 0.04028451 0.0010137099
## 181   181 0.03232535 0.1922188 0.02478443 0.001752774 0.05032878 0.0011810224
## 182   182 0.03206500 0.2051892 0.02458547 0.001743100 0.04014883 0.0010145327
## 183   183 0.03232765 0.1921171 0.02478468 0.001749040 0.05030386 0.0011837940
## 184   184 0.03206652 0.2051432 0.02458384 0.001742172 0.04007395 0.0010134162
## 185   185 0.03234190 0.1917588 0.02481271 0.001830687 0.05387810 0.0011521150
## 186   186 0.03227329 0.1967692 0.02478257 0.002317780 0.06271779 0.0015669544
## 187   187 0.03206462 0.2052230 0.02457848 0.001738596 0.03992543 0.0010106557
## 188   188 0.03232442 0.1922840 0.02477902 0.001750168 0.05040209 0.0011869352
## 189   189 0.03206730 0.2051053 0.02458020 0.001741748 0.04005787 0.0010148205
## 190   190 0.03268609 0.1740963 0.02505727 0.001904002 0.05883026 0.0011996750
## 191   191 0.03206811 0.2050694 0.02458214 0.001743898 0.04008077 0.0010127574
## 192   192 0.03207025 0.2049907 0.02458290 0.001744863 0.04009111 0.0010145145
## 193   193 0.03226731 0.1951171 0.02473239 0.001692544 0.03829879 0.0009779075
## 194   194 0.03222126 0.1970422 0.02467817 0.001950298 0.05525669 0.0011745391
## 195   195 0.03206803 0.2050948 0.02457807 0.001743314 0.04006339 0.0010159525
## 196   196 0.03235885 0.1911123 0.02482214 0.001840942 0.05463615 0.0011672515
## 197   197 0.03235678 0.1912217 0.02482040 0.001842818 0.05465493 0.0011682310
## 198   198 0.03271584 0.1719899 0.02510708 0.001697437 0.05321777 0.0011344318
## 199   199 0.03206995 0.2050216 0.02458109 0.001746296 0.04016722 0.0010193117
## 200   200 0.03206855 0.2050847 0.02457934 0.001748313 0.04027412 0.0010197175
## 201   201 0.03206861 0.2050848 0.02457943 0.001748213 0.04022911 0.0010208412
## 202   202 0.03224647 0.1954237 0.02474958 0.001826542 0.05292968 0.0010892217
## 203   203 0.03254725 0.1813046 0.02496030 0.001801565 0.05885501 0.0012624593
## 204   204 0.03206770 0.2051181 0.02457536 0.001745263 0.04015378 0.0010182995
## 205   205 0.03206632 0.2051692 0.02457445 0.001742740 0.04006480 0.0010177920
## 206   206 0.03206777 0.2051166 0.02457590 0.001743580 0.04009384 0.0010194146
## 207   207 0.03206837 0.2050758 0.02457590 0.001741224 0.04001211 0.0010176408
## 208   208 0.03206968 0.2050207 0.02457615 0.001739870 0.03996805 0.0010155763
## 209   209 0.03206953 0.2050276 0.02457688 0.001738998 0.03991646 0.0010149879
## 210   210 0.03225028 0.1952557 0.02475096 0.001818533 0.05277656 0.0010845038
## 211   211 0.03207051 0.2049941 0.02457788 0.001736662 0.03982307 0.0010140456
## 212   212 0.03243601 0.1863240 0.02485060 0.001611372 0.03934622 0.0009440440
## 213   213 0.03207192 0.2049312 0.02457959 0.001734728 0.03977390 0.0010133716
## 214   214 0.03223183 0.1965386 0.02468944 0.001680874 0.04282380 0.0009941573
## 215   215 0.03207362 0.2048447 0.02458269 0.001731383 0.03964952 0.0010114171
## 216   216 0.03273763 0.1719632 0.02515936 0.001930451 0.05571995 0.0012589511
## 217   217 0.03207360 0.2048493 0.02458291 0.001730862 0.03962953 0.0010117337
## 218   218 0.03207367 0.2048519 0.02458347 0.001732159 0.03967375 0.0010137218
## 219   219 0.03235576 0.1909350 0.02478746 0.001760052 0.05222916 0.0012001769
## 220   220 0.03227708 0.1947009 0.02475990 0.001801636 0.05148695 0.0011190900
## 221   221 0.03207300 0.2048841 0.02458170 0.001731904 0.03966357 0.0010147819
## 222   222 0.03230902 0.1927392 0.02476008 0.001671419 0.05003771 0.0009861776
## 223   223 0.03207242 0.2049032 0.02458086 0.001732350 0.03968642 0.0010157095
## 224   224 0.03207233 0.2049031 0.02458054 0.001731883 0.03966506 0.0010149805
## 225   225 0.03227920 0.1946185 0.02476111 0.001804842 0.05173172 0.0011225555
## 226   226 0.03207187 0.2049254 0.02458114 0.001731322 0.03964074 0.0010143681
## 227   227 0.03207190 0.2049216 0.02458136 0.001730803 0.03962087 0.0010141621
## 228   228 0.03207191 0.2049236 0.02458172 0.001730910 0.03963172 0.0010141598
## 229   229 0.03207167 0.2049328 0.02458197 0.001730707 0.03962512 0.0010136789
## 230   230 0.03230080 0.1960977 0.02479123 0.002369198 0.06353051 0.0015907885
## 231   231 0.03231540 0.1930192 0.02481303 0.001976288 0.05295142 0.0012767256
## 232   232 0.03207141 0.2049454 0.02458210 0.001730423 0.03961255 0.0010130997
## 233   233 0.03207113 0.2049573 0.02458187 0.001730504 0.03961599 0.0010128952
## 234   234 0.03260853 0.1782309 0.02497782 0.001663699 0.05826799 0.0011485919
## 235   235 0.03227027 0.1945174 0.02477004 0.001828469 0.05397957 0.0010990078
## 236   236 0.03228877 0.1940785 0.02475450 0.001688007 0.03919501 0.0009894918
## 237   237 0.03315800 0.1499859 0.02548972 0.001657879 0.05323616 0.0010000809
## 238   238 0.03365107 0.1256566 0.02588090 0.001340379 0.04268967 0.0008880158
## 239   239 0.03301288 0.1606469 0.02538267 0.002313877 0.07044983 0.0016553371
## 240   240 0.03207168 0.2049328 0.02458231 0.001730399 0.03960843 0.0010134866
## [1] "Best Model"
##   nvmax
## 9     9

## `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.

## [1] "Coefficients of final model:"
##                  Estimate         2.5 %        97.5 %
## (Intercept)  1.997430e+00  1.990899e+00  2.003962e+00
## x4          -5.115588e-05 -6.853808e-05 -3.377368e-05
## x7           1.102188e-02  9.794023e-03  1.224974e-02
## x9           3.070804e-03  2.432449e-03  3.709159e-03
## x10          1.278875e-03  6.873502e-04  1.870401e-03
## x16          9.700205e-04  5.568889e-04  1.383152e-03
## x17          1.600779e-03  9.763881e-04  2.225170e-03
## stat98       3.343631e-03  2.875903e-03  3.811359e-03
## stat110     -3.137873e-03 -3.613099e-03 -2.662647e-03
## x18.sqrt     2.631786e-02  2.450088e-02  2.813484e-02

Test

if (algo.stepwise.caret == TRUE){
  test.model(model.stepwise, data.test
             ,method = 'leapSeq',subopt = NULL
             ,formula = formula, feature.names = feature.names, label.names = label.names
             ,id = id
             ,draw.limits = TRUE, transformation = t)
  
}

## [1] "Summary of predicted values: "
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##   2.044   2.084   2.097   2.096   2.108   2.145 
## [1] "leapSeq  Test MSE: 0.00104102201936567"

LASSO with CV

Train

if (algo.LASSO.caret == TRUE){
  set.seed(1)
  tune.grid= expand.grid(alpha = 1,lambda = 10^seq(from=-4,to=-2,length=100))
  returned = train.caret.glmselect(formula = formula
                                   ,data =  data.train
                                   ,method = "glmnet"
                                   ,subopt = 'LASSO'
                                   ,tune.grid = tune.grid
                                   ,feature.names = feature.names)
  model.LASSO.caret = returned$model
}

## Aggregating results
## Selecting tuning parameters
## Fitting alpha = 1, lambda = 0.000534 on full training set
## glmnet 
## 
## 5584 samples
##  240 predictor
## 
## No pre-processing
## Resampling: Cross-Validated (10 fold) 
## Summary of sample sizes: 5026, 5026, 5026, 5025, 5025, 5026, ... 
## Resampling results across tuning parameters:
## 
##   lambda        RMSE        Rsquared   MAE       
##   0.0001000000  0.03182870  0.2137076  0.02441023
##   0.0001047616  0.03181915  0.2140763  0.02440355
##   0.0001097499  0.03180927  0.2144604  0.02439658
##   0.0001149757  0.03179912  0.2148574  0.02438943
##   0.0001204504  0.03178873  0.2152657  0.02438212
##   0.0001261857  0.03177804  0.2156889  0.02437454
##   0.0001321941  0.03176718  0.2161211  0.02436676
##   0.0001384886  0.03175605  0.2165672  0.02435884
##   0.0001450829  0.03174469  0.2170260  0.02435083
##   0.0001519911  0.03173310  0.2174974  0.02434271
##   0.0001592283  0.03172105  0.2179927  0.02433422
##   0.0001668101  0.03170877  0.2185026  0.02432563
##   0.0001747528  0.03169627  0.2190254  0.02431677
##   0.0001830738  0.03168364  0.2195589  0.02430775
##   0.0001917910  0.03167103  0.2200965  0.02429872
##   0.0002009233  0.03165835  0.2206429  0.02428958
##   0.0002104904  0.03164552  0.2212024  0.02428038
##   0.0002205131  0.03163245  0.2217799  0.02427113
##   0.0002310130  0.03161944  0.2223626  0.02426193
##   0.0002420128  0.03160645  0.2229523  0.02425284
##   0.0002535364  0.03159361  0.2235431  0.02424387
##   0.0002656088  0.03158092  0.2241359  0.02423522
##   0.0002782559  0.03156877  0.2247131  0.02422727
##   0.0002915053  0.03155682  0.2252902  0.02421969
##   0.0003053856  0.03154558  0.2258456  0.02421276
##   0.0003199267  0.03153478  0.2263890  0.02420621
##   0.0003351603  0.03152471  0.2269062  0.02419982
##   0.0003511192  0.03151528  0.2274023  0.02419384
##   0.0003678380  0.03150698  0.2278516  0.02418917
##   0.0003853529  0.03149941  0.2282761  0.02418525
##   0.0004037017  0.03149331  0.2286417  0.02418283
##   0.0004229243  0.03148794  0.2289830  0.02418114
##   0.0004430621  0.03148344  0.2292964  0.02418039
##   0.0004641589  0.03147993  0.2295734  0.02418032
##   0.0004862602  0.03147724  0.2298251  0.02418196
##   0.0005094138  0.03147584  0.2300250  0.02418469
##   0.0005336699  0.03147529  0.2301977  0.02418786
##   0.0005590810  0.03147632  0.2303033  0.02419231
##   0.0005857021  0.03147903  0.2303349  0.02419858
##   0.0006135907  0.03148305  0.2303133  0.02420609
##   0.0006428073  0.03148930  0.2301906  0.02421507
##   0.0006734151  0.03149671  0.2300230  0.02422495
##   0.0007054802  0.03150530  0.2298123  0.02423585
##   0.0007390722  0.03151517  0.2295490  0.02424820
##   0.0007742637  0.03152577  0.2292579  0.02426172
##   0.0008111308  0.03153760  0.2289201  0.02427645
##   0.0008497534  0.03154991  0.2285769  0.02429172
##   0.0008902151  0.03156351  0.2281859  0.02430795
##   0.0009326033  0.03157678  0.2278436  0.02432303
##   0.0009770100  0.03159157  0.2274433  0.02433921
##   0.0010235310  0.03160721  0.2270216  0.02435577
##   0.0010722672  0.03162495  0.2265139  0.02437466
##   0.0011233240  0.03164458  0.2259205  0.02439543
##   0.0011768120  0.03166596  0.2252600  0.02441787
##   0.0012328467  0.03168793  0.2246032  0.02444073
##   0.0012915497  0.03171161  0.2238829  0.02446527
##   0.0013530478  0.03173511  0.2232133  0.02449030
##   0.0014174742  0.03176079  0.2224605  0.02451735
##   0.0014849683  0.03178845  0.2216425  0.02454585
##   0.0015556761  0.03181891  0.2207066  0.02457706
##   0.0016297508  0.03185100  0.2197293  0.02460969
##   0.0017073526  0.03188570  0.2186501  0.02464475
##   0.0017886495  0.03192157  0.2175594  0.02468123
##   0.0018738174  0.03195953  0.2164084  0.02471929
##   0.0019630407  0.03199700  0.2153667  0.02475724
##   0.0020565123  0.03203688  0.2142507  0.02479680
##   0.0021544347  0.03207493  0.2133450  0.02483538
##   0.0022570197  0.03211587  0.2123547  0.02487609
##   0.0023644894  0.03215811  0.2114023  0.02491772
##   0.0024770764  0.03220356  0.2103639  0.02496151
##   0.0025950242  0.03225103  0.2093330  0.02500656
##   0.0027185882  0.03230286  0.2081536  0.02505555
##   0.0028480359  0.03235951  0.2067936  0.02510779
##   0.0029836472  0.03242153  0.2052091  0.02516370
##   0.0031257158  0.03248944  0.2033564  0.02522413
##   0.0032745492  0.03256378  0.2011821  0.02528939
##   0.0034304693  0.03264514  0.1986212  0.02535920
##   0.0035938137  0.03273272  0.1957147  0.02543439
##   0.0037649358  0.03282239  0.1927831  0.02551137
##   0.0039442061  0.03291738  0.1895721  0.02559210
##   0.0041320124  0.03300975  0.1867804  0.02567115
##   0.0043287613  0.03310995  0.1835006  0.02575558
##   0.0045348785  0.03321914  0.1795815  0.02584532
##   0.0047508102  0.03333853  0.1748038  0.02594294
##   0.0049770236  0.03346904  0.1689546  0.02604857
##   0.0052140083  0.03361092  0.1618476  0.02616254
##   0.0054622772  0.03375718  0.1540228  0.02627820
##   0.0057223677  0.03390841  0.1454038  0.02639513
##   0.0059948425  0.03402682  0.1400307  0.02648161
##   0.0062802914  0.03414613  0.1344591  0.02656781
##   0.0065793322  0.03425881  0.1296572  0.02664738
##   0.0068926121  0.03437990  0.1237110  0.02673245
##   0.0072208090  0.03449188  0.1185498  0.02680925
##   0.0075646333  0.03460250  0.1136746  0.02688495
##   0.0079248290  0.03468582  0.1132974  0.02693932
##   0.0083021757  0.03477462  0.1132051  0.02699714
##   0.0086974900  0.03487128  0.1132051  0.02706083
##   0.0091116276  0.03497705  0.1132051  0.02713128
##   0.0095454846  0.03509275  0.1132051  0.02720824
##   0.0100000000  0.03521928  0.1132051  0.02729254
## 
## Tuning parameter 'alpha' was held constant at a value of 1
## RMSE was used to select the optimal model using the smallest value.
## The final values used for the model were alpha = 1 and lambda = 0.0005336699.

##    alpha       lambda
## 37     1 0.0005336699
##     alpha       lambda       RMSE  Rsquared        MAE      RMSESD RsquaredSD        MAESD
## 1       1 0.0001000000 0.03182870 0.2137076 0.02441023 0.001693146 0.03945645 0.0009800245
## 2       1 0.0001047616 0.03181915 0.2140763 0.02440355 0.001690922 0.03942317 0.0009785114
## 3       1 0.0001097499 0.03180927 0.2144604 0.02439658 0.001688426 0.03938261 0.0009767729
## 4       1 0.0001149757 0.03179912 0.2148574 0.02438943 0.001685800 0.03933841 0.0009749356
## 5       1 0.0001204504 0.03178873 0.2152657 0.02438212 0.001683062 0.03928942 0.0009730537
## 6       1 0.0001261857 0.03177804 0.2156889 0.02437454 0.001680227 0.03923822 0.0009711118
## 7       1 0.0001321941 0.03176718 0.2161211 0.02436676 0.001677332 0.03918452 0.0009691814
## 8       1 0.0001384886 0.03175605 0.2165672 0.02435884 0.001674263 0.03912663 0.0009671920
## 9       1 0.0001450829 0.03174469 0.2170260 0.02435083 0.001670837 0.03905908 0.0009654273
## 10      1 0.0001519911 0.03173310 0.2174974 0.02434271 0.001667244 0.03898763 0.0009637174
## 11      1 0.0001592283 0.03172105 0.2179927 0.02433422 0.001663569 0.03891894 0.0009621797
## 12      1 0.0001668101 0.03170877 0.2185026 0.02432563 0.001659731 0.03884576 0.0009605895
## 13      1 0.0001747528 0.03169627 0.2190254 0.02431677 0.001655845 0.03877402 0.0009589892
## 14      1 0.0001830738 0.03168364 0.2195589 0.02430775 0.001651766 0.03869709 0.0009572153
## 15      1 0.0001917910 0.03167103 0.2200965 0.02429872 0.001647780 0.03862923 0.0009555977
## 16      1 0.0002009233 0.03165835 0.2206429 0.02428958 0.001643601 0.03855677 0.0009538373
## 17      1 0.0002104904 0.03164552 0.2212024 0.02428038 0.001639412 0.03848927 0.0009521441
## 18      1 0.0002205131 0.03163245 0.2217799 0.02427113 0.001635170 0.03842442 0.0009504364
## 19      1 0.0002310130 0.03161944 0.2223626 0.02426193 0.001631188 0.03837580 0.0009492318
## 20      1 0.0002420128 0.03160645 0.2229523 0.02425284 0.001627167 0.03832757 0.0009480802
## 21      1 0.0002535364 0.03159361 0.2235431 0.02424387 0.001623645 0.03830226 0.0009470889
## 22      1 0.0002656088 0.03158092 0.2241359 0.02423522 0.001619988 0.03827442 0.0009460304
## 23      1 0.0002782559 0.03156877 0.2247131 0.02422727 0.001616684 0.03826799 0.0009452206
## 24      1 0.0002915053 0.03155682 0.2252902 0.02421969 0.001613226 0.03825975 0.0009442282
## 25      1 0.0003053856 0.03154558 0.2258456 0.02421276 0.001610379 0.03829091 0.0009433200
## 26      1 0.0003199267 0.03153478 0.2263890 0.02420621 0.001607295 0.03831571 0.0009421721
## 27      1 0.0003351603 0.03152471 0.2269062 0.02419982 0.001604161 0.03834123 0.0009413149
## 28      1 0.0003511192 0.03151528 0.2274023 0.02419384 0.001600953 0.03836436 0.0009408317
## 29      1 0.0003678380 0.03150698 0.2278516 0.02418917 0.001597607 0.03836861 0.0009401939
## 30      1 0.0003853529 0.03149941 0.2282761 0.02418525 0.001594108 0.03836713 0.0009392682
## 31      1 0.0004037017 0.03149331 0.2286417 0.02418283 0.001590996 0.03837797 0.0009383324
## 32      1 0.0004229243 0.03148794 0.2289830 0.02418114 0.001587791 0.03838941 0.0009372545
## 33      1 0.0004430621 0.03148344 0.2292964 0.02418039 0.001584576 0.03840897 0.0009361478
## 34      1 0.0004641589 0.03147993 0.2295734 0.02418032 0.001581207 0.03842766 0.0009352930
## 35      1 0.0004862602 0.03147724 0.2298251 0.02418196 0.001577649 0.03842867 0.0009340461
## 36      1 0.0005094138 0.03147584 0.2300250 0.02418469 0.001573613 0.03840765 0.0009323596
## 37      1 0.0005336699 0.03147529 0.2301977 0.02418786 0.001569060 0.03835303 0.0009303565
## 38      1 0.0005590810 0.03147632 0.2303033 0.02419231 0.001563847 0.03826411 0.0009275146
## 39      1 0.0005857021 0.03147903 0.2303349 0.02419858 0.001557618 0.03810712 0.0009237975
## 40      1 0.0006135907 0.03148305 0.2303133 0.02420609 0.001551063 0.03793644 0.0009197778
## 41      1 0.0006428073 0.03148930 0.2301906 0.02421507 0.001544691 0.03777284 0.0009163365
## 42      1 0.0006734151 0.03149671 0.2300230 0.02422495 0.001538400 0.03761956 0.0009132476
## 43      1 0.0007054802 0.03150530 0.2298123 0.02423585 0.001533169 0.03753338 0.0009102606
## 44      1 0.0007390722 0.03151517 0.2295490 0.02424820 0.001527820 0.03744787 0.0009068783
## 45      1 0.0007742637 0.03152577 0.2292579 0.02426172 0.001522240 0.03736032 0.0009035369
## 46      1 0.0008111308 0.03153760 0.2289201 0.02427645 0.001516688 0.03728506 0.0009000239
## 47      1 0.0008497534 0.03154991 0.2285769 0.02429172 0.001511755 0.03725715 0.0008963947
## 48      1 0.0008902151 0.03156351 0.2281859 0.02430795 0.001506507 0.03722456 0.0008928501
## 49      1 0.0009326033 0.03157678 0.2278436 0.02432303 0.001501021 0.03720320 0.0008886619
## 50      1 0.0009770100 0.03159157 0.2274433 0.02433921 0.001495256 0.03716526 0.0008841318
## 51      1 0.0010235310 0.03160721 0.2270216 0.02435577 0.001489218 0.03709465 0.0008798247
## 52      1 0.0010722672 0.03162495 0.2265139 0.02437466 0.001482840 0.03700472 0.0008753560
## 53      1 0.0011233240 0.03164458 0.2259205 0.02439543 0.001476820 0.03691704 0.0008712574
## 54      1 0.0011768120 0.03166596 0.2252600 0.02441787 0.001470614 0.03683378 0.0008677339
## 55      1 0.0012328467 0.03168793 0.2246032 0.02444073 0.001464015 0.03675448 0.0008659036
## 56      1 0.0012915497 0.03171161 0.2238829 0.02446527 0.001457104 0.03666595 0.0008639256
## 57      1 0.0013530478 0.03173511 0.2232133 0.02449030 0.001449634 0.03656470 0.0008614688
## 58      1 0.0014174742 0.03176079 0.2224605 0.02451735 0.001441737 0.03644506 0.0008584619
## 59      1 0.0014849683 0.03178845 0.2216425 0.02454585 0.001432974 0.03627258 0.0008544587
## 60      1 0.0015556761 0.03181891 0.2207066 0.02457706 0.001423400 0.03605076 0.0008499458
## 61      1 0.0016297508 0.03185100 0.2197293 0.02460969 0.001413036 0.03580356 0.0008448008
## 62      1 0.0017073526 0.03188570 0.2186501 0.02464475 0.001401803 0.03551682 0.0008388329
## 63      1 0.0017886495 0.03192157 0.2175594 0.02468123 0.001390078 0.03520147 0.0008328110
## 64      1 0.0018738174 0.03195953 0.2164084 0.02471929 0.001378422 0.03489973 0.0008265072
## 65      1 0.0019630407 0.03199700 0.2153667 0.02475724 0.001367349 0.03459826 0.0008197790
## 66      1 0.0020565123 0.03203688 0.2142507 0.02479680 0.001356725 0.03432457 0.0008135705
## 67      1 0.0021544347 0.03207493 0.2133450 0.02483538 0.001347811 0.03407788 0.0008092411
## 68      1 0.0022570197 0.03211587 0.2123547 0.02487609 0.001338615 0.03381116 0.0008048197
## 69      1 0.0023644894 0.03215811 0.2114023 0.02491772 0.001329824 0.03354756 0.0008015722
## 70      1 0.0024770764 0.03220356 0.2103639 0.02496151 0.001320981 0.03329859 0.0007979716
## 71      1 0.0025950242 0.03225103 0.2093330 0.02500656 0.001311728 0.03306585 0.0007946204
## 72      1 0.0027185882 0.03230286 0.2081536 0.02505555 0.001302082 0.03281275 0.0007914727
## 73      1 0.0028480359 0.03235951 0.2067936 0.02510779 0.001292026 0.03253696 0.0007883657
## 74      1 0.0029836472 0.03242153 0.2052091 0.02516370 0.001281506 0.03223108 0.0007850475
## 75      1 0.0031257158 0.03248944 0.2033564 0.02522413 0.001270512 0.03189223 0.0007813295
## 76      1 0.0032745492 0.03256378 0.2011821 0.02528939 0.001259030 0.03151684 0.0007774595
## 77      1 0.0034304693 0.03264514 0.1986212 0.02535920 0.001247045 0.03110118 0.0007734591
## 78      1 0.0035938137 0.03273272 0.1957147 0.02543439 0.001235635 0.03079069 0.0007686966
## 79      1 0.0037649358 0.03282239 0.1927831 0.02551137 0.001224075 0.03047186 0.0007623563
## 80      1 0.0039442061 0.03291738 0.1895721 0.02559210 0.001213383 0.03031859 0.0007564518
## 81      1 0.0041320124 0.03300975 0.1867804 0.02567115 0.001202377 0.03010563 0.0007491561
## 82      1 0.0043287613 0.03310995 0.1835006 0.02575558 0.001190824 0.02986572 0.0007404788
## 83      1 0.0045348785 0.03321914 0.1795815 0.02584532 0.001178684 0.02966345 0.0007312001
## 84      1 0.0047508102 0.03333853 0.1748038 0.02594294 0.001166087 0.02947972 0.0007215574
## 85      1 0.0049770236 0.03346904 0.1689546 0.02604857 0.001153037 0.02932342 0.0007116703
## 86      1 0.0052140083 0.03361092 0.1618476 0.02616254 0.001140040 0.02923709 0.0007029476
## 87      1 0.0054622772 0.03375718 0.1540228 0.02627820 0.001129102 0.02889451 0.0006960095
## 88      1 0.0057223677 0.03390841 0.1454038 0.02639513 0.001120906 0.02951566 0.0006910245
## 89      1 0.0059948425 0.03402682 0.1400307 0.02648161 0.001110666 0.02868495 0.0006806742
## 90      1 0.0062802914 0.03414613 0.1344591 0.02656781 0.001101971 0.02808243 0.0006700052
## 91      1 0.0065793322 0.03425881 0.1296572 0.02664738 0.001090611 0.02792209 0.0006585626
## 92      1 0.0068926121 0.03437990 0.1237110 0.02673245 0.001079217 0.02776121 0.0006478933
## 93      1 0.0072208090 0.03449188 0.1185498 0.02680925 0.001072559 0.02616045 0.0006385552
## 94      1 0.0075646333 0.03460250 0.1136746 0.02688495 0.001069767 0.02587502 0.0006324191
## 95      1 0.0079248290 0.03468582 0.1132974 0.02693932 0.001063775 0.02585808 0.0006273047
## 96      1 0.0083021757 0.03477462 0.1132051 0.02699714 0.001057784 0.02571127 0.0006219403
## 97      1 0.0086974900 0.03487128 0.1132051 0.02706083 0.001052059 0.02571127 0.0006175194
## 98      1 0.0091116276 0.03497705 0.1132051 0.02713128 0.001046273 0.02571127 0.0006133177
## 99      1 0.0095454846 0.03509275 0.1132051 0.02720824 0.001040454 0.02571127 0.0006101350
## 100     1 0.0100000000 0.03521928 0.1132051 0.02729254 0.001034631 0.02571127 0.0006080836

## `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.

## [1] "Coefficients"
##                model.coef
## (Intercept)  1.994914e+00
## x4          -4.002366e-05
## x7           1.024848e-02
## x8           2.632932e-04
## x9           2.657880e-03
## x10          9.133035e-04
## x11          6.455140e+04
## x16          7.020978e-04
## x17          1.191654e-03
## x19          5.520463e-05
## x21          7.165645e-05
## x22         -5.468921e-05
## stat4       -2.637319e-04
## stat5       -7.989719e-05
## stat8        4.810475e-06
## stat10      -2.798960e-05
## stat13      -5.312046e-05
## stat14      -5.253446e-04
## stat15      -1.651295e-04
## stat20      -2.894415e-05
## stat22      -2.008800e-04
## stat23       4.078054e-04
## stat24      -2.411983e-04
## stat25      -1.917754e-04
## stat30       4.191991e-05
## stat35      -1.717092e-04
## stat37      -2.508905e-04
## stat38       2.179254e-04
## stat41      -3.531813e-04
## stat45      -1.414883e-05
## stat54      -9.071240e-05
## stat59       1.036277e-04
## stat60       2.073650e-04
## stat65      -6.888327e-05
## stat82       2.532324e-05
## stat91      -6.976596e-05
## stat92      -4.683807e-05
## stat96      -1.531965e-04
## stat98       3.040515e-03
## stat99       6.912739e-05
## stat100      1.716283e-04
## stat103     -2.060162e-04
## stat106     -7.235702e-05
## stat110     -2.881217e-03
## stat113     -1.179757e-04
## stat118     -1.133401e-04
## stat119      9.665617e-07
## stat121     -4.413637e-06
## stat144      3.607457e-04
## stat146     -3.923816e-05
## stat147     -1.852727e-05
## stat148     -7.355995e-05
## stat149     -1.424650e-04
## stat156      2.644802e-04
## stat164      4.808845e-05
## stat168     -3.883916e-05
## stat195      7.705825e-05
## stat198     -1.183566e-04
## stat204     -2.480254e-04
## stat207      4.379853e-05
## x18.sqrt     2.509646e-02

Test

if (algo.LASSO.caret == TRUE){
  test.model(model.LASSO.caret, data.test
             ,method = 'glmnet',subopt = "LASSO"
             ,formula = formula, feature.names = feature.names, label.names = label.names
             ,draw.limits = TRUE, transformation = t)
}

## [1] "Summary of predicted values: "
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##   2.045   2.085   2.097   2.096   2.107   2.141 
## [1] "glmnet LASSO Test MSE: 0.00104517306987558"

LARS with CV

Train

if (algo.LARS.caret == TRUE){
  set.seed(1)
  returned = train.caret.glmselect(formula = formula
                                   ,data =  data.train
                                   ,method = "lars"
                                   ,subopt = 'NULL'
                                   ,feature.names = feature.names)
  model.LARS.caret = returned$model
}

## Warning in nominalTrainWorkflow(x = x, y = y, wts = weights, info = trainInfo, : There were missing values in resampled
## performance measures.

## Aggregating results
## Selecting tuning parameters
## Fitting fraction = 0.404 on full training set
## Least Angle Regression 
## 
## 5584 samples
##  240 predictor
## 
## Pre-processing: centered (240), scaled (240) 
## Resampling: Cross-Validated (10 fold) 
## Summary of sample sizes: 5026, 5026, 5026, 5025, 5025, 5026, ... 
## Resampling results across tuning parameters:
## 
##   fraction    RMSE        Rsquared   MAE       
##   0.00000000  0.03582413        NaN  0.02770080
##   0.01010101  0.03540595  0.1132051  0.02741614
##   0.02020202  0.03503208  0.1132051  0.02716554
##   0.03030303  0.03470397  0.1132051  0.02694770
##   0.04040404  0.03442783  0.1210066  0.02676414
##   0.05050505  0.03417085  0.1334345  0.02658317
##   0.06060606  0.03394463  0.1433609  0.02642105
##   0.07070707  0.03372956  0.1553607  0.02625582
##   0.08080808  0.03352388  0.1664433  0.02609206
##   0.09090909  0.03333009  0.1752792  0.02593508
##   0.10101010  0.03314838  0.1822585  0.02578632
##   0.11111111  0.03297905  0.1877397  0.02564435
##   0.12121212  0.03282549  0.1924615  0.02551380
##   0.13131313  0.03267923  0.1975846  0.02538996
##   0.14141414  0.03254236  0.2018999  0.02527278
##   0.15151515  0.03241552  0.2054266  0.02516078
##   0.16161616  0.03229883  0.2082933  0.02505421
##   0.17171717  0.03219309  0.2106309  0.02495408
##   0.18181818  0.03210235  0.2127016  0.02486571
##   0.19191919  0.03202232  0.2146368  0.02478477
##   0.20202020  0.03194921  0.2167190  0.02471080
##   0.21212121  0.03188044  0.2187742  0.02464139
##   0.22222222  0.03181521  0.2207958  0.02457495
##   0.23232323  0.03175799  0.2225278  0.02451541
##   0.24242424  0.03171064  0.2238902  0.02446561
##   0.25252525  0.03166878  0.2251593  0.02442188
##   0.26262626  0.03163238  0.2262811  0.02438391
##   0.27272727  0.03160361  0.2271049  0.02435351
##   0.28282828  0.03158123  0.2277074  0.02432900
##   0.29292929  0.03156220  0.2282257  0.02430725
##   0.30303030  0.03154560  0.2286958  0.02428701
##   0.31313131  0.03153132  0.2290834  0.02426925
##   0.32323232  0.03151874  0.2294364  0.02425372
##   0.33333333  0.03150815  0.2297192  0.02424055
##   0.34343434  0.03149904  0.2299514  0.02422915
##   0.35353535  0.03149054  0.2301770  0.02421826
##   0.36363636  0.03148428  0.2303104  0.02420917
##   0.37373737  0.03147971  0.2303783  0.02420182
##   0.38383838  0.03147689  0.2303738  0.02419595
##   0.39393939  0.03147600  0.2302808  0.02419190
##   0.40404040  0.03147557  0.2301769  0.02418866
##   0.41414141  0.03147607  0.2300369  0.02418619
##   0.42424242  0.03147741  0.2298669  0.02418400
##   0.43434343  0.03147914  0.2296844  0.02418259
##   0.44444444  0.03148085  0.2295107  0.02418113
##   0.45454545  0.03148324  0.2293129  0.02418058
##   0.46464646  0.03148640  0.2290873  0.02418091
##   0.47474747  0.03149027  0.2288332  0.02418160
##   0.48484848  0.03149424  0.2285836  0.02418302
##   0.49494949  0.03149875  0.2283138  0.02418507
##   0.50505051  0.03150418  0.2280050  0.02418785
##   0.51515152  0.03151039  0.2276631  0.02419141
##   0.52525253  0.03151728  0.2272942  0.02419569
##   0.53535354  0.03152455  0.2269138  0.02420033
##   0.54545455  0.03153195  0.2265324  0.02420515
##   0.55555556  0.03153998  0.2261271  0.02421026
##   0.56565657  0.03154869  0.2256931  0.02421572
##   0.57575758  0.03155724  0.2252730  0.02422098
##   0.58585859  0.03156620  0.2248398  0.02422660
##   0.59595960  0.03157543  0.2243990  0.02423244
##   0.60606061  0.03158494  0.2239508  0.02423867
##   0.61616162  0.03159473  0.2234949  0.02424538
##   0.62626263  0.03160487  0.2230269  0.02425251
##   0.63636364  0.03161523  0.2225546  0.02425977
##   0.64646465  0.03162553  0.2220903  0.02426695
##   0.65656566  0.03163604  0.2216219  0.02427441
##   0.66666667  0.03164682  0.2211470  0.02428211
##   0.67676768  0.03165759  0.2206775  0.02428968
##   0.68686869  0.03166844  0.2202084  0.02429735
##   0.69696970  0.03167956  0.2197316  0.02430527
##   0.70707071  0.03169082  0.2192540  0.02431321
##   0.71717172  0.03170235  0.2187690  0.02432144
##   0.72727273  0.03171414  0.2182764  0.02432963
##   0.73737374  0.03172599  0.2177857  0.02433783
##   0.74747475  0.03173779  0.2173025  0.02434609
##   0.75757576  0.03174967  0.2168209  0.02435432
##   0.76767677  0.03176159  0.2163421  0.02436268
##   0.77777778  0.03177379  0.2158555  0.02437139
##   0.78787879  0.03178607  0.2153697  0.02438020
##   0.79797980  0.03179849  0.2148808  0.02438895
##   0.80808081  0.03181113  0.2143863  0.02439782
##   0.81818182  0.03182379  0.2138953  0.02440678
##   0.82828283  0.03183650  0.2134065  0.02441560
##   0.83838384  0.03184946  0.2129117  0.02442461
##   0.84848485  0.03186265  0.2124109  0.02443386
##   0.85858586  0.03187583  0.2119155  0.02444299
##   0.86868687  0.03188914  0.2114178  0.02445240
##   0.87878788  0.03190257  0.2109196  0.02446203
##   0.88888889  0.03191625  0.2104145  0.02447182
##   0.89898990  0.03193014  0.2099052  0.02448164
##   0.90909091  0.03194389  0.2094070  0.02449128
##   0.91919192  0.03195769  0.2089116  0.02450091
##   0.92929293  0.03197160  0.2084148  0.02451068
##   0.93939394  0.03198565  0.2079157  0.02452050
##   0.94949495  0.03199982  0.2074155  0.02453046
##   0.95959596  0.03201395  0.2069219  0.02454039
##   0.96969697  0.03202814  0.2064297  0.02455073
##   0.97979798  0.03204240  0.2059379  0.02456118
##   0.98989899  0.03205687  0.2054404  0.02457160
##   1.00000000  0.03207168  0.2049328  0.02458231
## 
## RMSE was used to select the optimal model using the smallest value.
## The final value used for the model was fraction = 0.4040404.

##     fraction
## 41 0.4040404

## Warning: Removed 1 rows containing missing values (geom_point).

## `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.

## [1] "Coefficients"
##            x4            x7            x8            x9           x10           x11           x16           x17 
## -1.895935e-03  6.888954e-03  7.569258e-04  3.434031e-03  1.269481e-03  3.676435e-04  1.398042e-03  1.569943e-03 
##           x19           x21           x22         stat4         stat5        stat10        stat13        stat14 
##  1.345776e-04  7.218064e-04 -6.144920e-05 -4.475185e-04 -1.338374e-04 -4.190862e-05 -8.605246e-05 -9.037538e-04 
##        stat15        stat20        stat22        stat23        stat24        stat25        stat30        stat35 
## -2.823696e-04 -4.370507e-05 -3.404304e-04  7.018155e-04 -4.102895e-04 -3.264800e-04  6.587527e-05 -2.895053e-04 
##        stat37        stat38        stat41        stat45        stat54        stat59        stat60        stat65 
## -4.231346e-04  3.686324e-04 -6.099668e-04 -1.833645e-05 -1.486452e-04  1.732624e-04  3.504797e-04 -1.121047e-04 
##        stat82        stat91        stat92        stat96        stat98        stat99       stat100       stat103 
##  3.725948e-05 -1.154440e-04 -7.436628e-05 -2.585713e-04  5.365784e-03  1.121694e-04  2.896916e-04 -3.438299e-04 
##       stat106       stat110       stat113       stat118       stat144       stat146       stat147       stat148 
## -1.197480e-04 -5.001610e-03 -1.969201e-04 -1.902694e-04  6.195373e-04 -6.053854e-05 -2.531191e-05 -1.216201e-04 
##       stat149       stat156       stat164       stat168       stat195       stat198       stat204       stat207 
## -2.378832e-04  4.447563e-04  7.603393e-05 -6.001526e-05  1.258098e-04 -1.978183e-04 -4.247122e-04  6.878251e-05 
##      x18.sqrt 
##  1.140877e-02

Test

if (algo.LARS.caret == TRUE){
  test.model(model.LARS.caret, data.test
             ,method = 'lars',subopt = NULL
             ,formula = formula, feature.names = feature.names, label.names = label.names
             ,draw.limits = TRUE, transformation = t)
}

## [1] "Summary of predicted values: "
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##   2.045   2.085   2.097   2.096   2.107   2.141 
## [1] "lars  Test MSE: 0.00104526315244053"

Session Info

sessionInfo()

## R version 3.5.2 (2018-12-20)
## Platform: x86_64-w64-mingw32/x64 (64-bit)
## Running under: Windows 10 x64 (build 17763)
## 
## Matrix products: default
## 
## locale:
## [1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252    LC_MONETARY=English_United States.1252
## [4] LC_NUMERIC=C                           LC_TIME=English_United States.1252    
## 
## attached base packages:
## [1] parallel  stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
##  [1] knitr_1.21                 htmltools_0.3.6            reshape2_1.4.3             lars_1.2                  
##  [5] doParallel_1.0.14          iterators_1.0.10           caret_6.0-81               leaps_3.0                 
##  [9] ggforce_0.1.3              rlist_0.4.6.1              car_3.0-2                  carData_3.0-2             
## [13] bestNormalize_1.3.0        scales_1.0.0               onewaytests_2.0            caTools_1.17.1.1          
## [17] mosaic_1.5.0               mosaicData_0.17.0          ggformula_0.9.1            ggstance_0.3.1            
## [21] lattice_0.20-38            DT_0.5                     ggiraphExtra_0.2.9         ggiraph_0.6.0             
## [25] investr_1.4.0              glmnet_2.0-16              foreach_1.4.4              Matrix_1.2-15             
## [29] MASS_7.3-51.1              PerformanceAnalytics_1.5.2 xts_0.11-2                 zoo_1.8-4                 
## [33] forcats_0.3.0              stringr_1.4.0              dplyr_0.8.0.1              purrr_0.3.0               
## [37] readr_1.3.1                tidyr_0.8.2                tibble_2.0.1               ggplot2_3.1.0             
## [41] tidyverse_1.2.1            usdm_1.1-18                raster_2.8-19              sp_1.3-1                  
## [45] pacman_0.5.0              
## 
## loaded via a namespace (and not attached):
##  [1] readxl_1.3.0       backports_1.1.3    plyr_1.8.4         lazyeval_0.2.1     splines_3.5.2      mycor_0.1.1       
##  [7] crosstalk_1.0.0    leaflet_2.0.2      digest_0.6.18      magrittr_1.5       mosaicCore_0.6.0   openxlsx_4.1.0    
## [13] recipes_0.1.4      modelr_0.1.3       gower_0.1.2        colorspace_1.4-0   rvest_0.3.2        ggrepel_0.8.0     
## [19] haven_2.0.0        xfun_0.4           crayon_1.3.4       jsonlite_1.6       survival_2.43-3    glue_1.3.0        
## [25] registry_0.5       gtable_0.2.0       ppcor_1.1          ipred_0.9-8        sjmisc_2.7.7       abind_1.4-5       
## [31] rngtools_1.3.1     bibtex_0.4.2       Rcpp_1.0.0         xtable_1.8-3       units_0.6-2        foreign_0.8-71    
## [37] stats4_3.5.2       lava_1.6.5         prodlim_2018.04.18 prediction_0.3.6.2 htmlwidgets_1.3    httr_1.4.0        
## [43] RColorBrewer_1.1-2 pkgconfig_2.0.2    farver_1.1.0       nnet_7.3-12        labeling_0.3       tidyselect_0.2.5  
## [49] rlang_0.3.1        later_0.8.0        munsell_0.5.0      cellranger_1.1.0   tools_3.5.2        cli_1.0.1         
## [55] generics_0.0.2     moments_0.14       sjlabelled_1.0.16  broom_0.5.1        evaluate_0.13      ggdendro_0.1-20   
## [61] yaml_2.2.0         ModelMetrics_1.2.2 zip_1.0.0          nlme_3.1-137       doRNG_1.7.1        mime_0.6          
## [67] xml2_1.2.0         compiler_3.5.2     rstudioapi_0.9.0   curl_3.3           tweenr_1.0.1       stringi_1.3.1     
## [73] highr_0.7          gdtools_0.1.7      stringdist_0.9.5.1 pillar_1.3.1       data.table_1.12.0  bitops_1.0-6      
## [79] httpuv_1.4.5.1     R6_2.4.0           promises_1.0.1     gridExtra_2.3      rio_0.5.16         codetools_0.2-15  
## [85] assertthat_0.2.0   pkgmaker_0.27      withr_2.1.2        nortest_1.0-4      mgcv_1.8-26        hms_0.4.2         
## [91] quadprog_1.5-5     grid_3.5.2         rpart_4.1-13       timeDate_3043.102  class_7.3-14       rmarkdown_1.11    
## [97] snakecase_0.9.2    shiny_1.2.0        lubridate_1.7.4

Master File

Joanna Duran, Nikhil Gupta, Max Moro

2019-02-16 11:22:50

User Inputs

Loading Data

Data validation

Output Variable

Histogram

QQPlot

Trasformation of Output Variable from y3 to y3.log

Best Normalizator y3

Predictors

Interesting Predictors

All Predictors

Histograms

Correlations

With Output Variable

Between All Variables

Scatter Plots with Output Variable

Multicollinearity - VIF

Feature Eng

Comparing Pre and Post Transformation Density Plots

Modeling

Train Test Split

Common Functions

Setup Formulae

Full Model

Checking with removal of high influence points

Grand Means Model

Variable Selection

Forward Selection with CV

Train

Test

Backward Elimination with CV

Train

Test

Stepwise Selection with CV

Train

Test

LASSO with CV

Train

Test

LARS with CV

Train

Test

Session Info